Signal processing device and signal processing method, and program and recording medium

ABSTRACT

A signal processing device, signal processing method, and program and recording medium which provide images and the like closer approximating real world signals. The signal processing device includes a processing region setting unit, a movement vector setting unit, a model generator, a normal equation generator, an activity detector, a weighting change unit, and a real world estimating unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.10/562,176, filed on Dec. 23, 2005, and is based upon and claims thebenefit of priority to International Application No. PCT/JP04/08691,filed on Jun. 15, 2004 and from the prior Japanese Patent ApplicationNo. 2003-184015 filed on Jun. 27, 2003. The entire contents of each ofthese documents are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to a signal processing device and signalprocessing method, and a program and recording medium, and in particularrelates to a signal processing device and signal processing method, anda program and recording medium, enabling images and the like with closerapproximation to real world signals.

BACKGROUND ART

Technology for detecting phenomena in the actual world (real world) withsensors and processing sampling data output from the sensors is widelyused. For example, image processing technology wherein the actual worldis imaged with an imaging sensor and sampling data which is the imagedata is processed, is widely employed.

Also, arrangements are known having second dimensions with fewerdimensions than first dimensions obtained by detecting with sensorsfirst signals, which are signals of the real world having firstdimensions, obtaining second signals including distortion as to thefirst signals, and performing signal processing based on the secondsignals, thereby generating third signals with alleviated distortion ascompared to the second signals (e.g., Japanese Unexamined PatentApplication Publication No. 2001-250119).

However, conventionally, signal processing taking into consideration thecontinuity of real world signals had not been performed, so obtainingimages and the like which closer approximate real world signals has beendifficult.

DISCLOSURE OF INVENTION

The present invention has been made in light of the situation asdescribed above, and provides for obtaining of images and the like whichcloser approximate real world signals.

A signal processing device according to the present invention comprises:processing region setting means for setting the processing region withindata, wherein light signals of the real world have been projected on aplurality of pixels each having time integration effects and a part ofthe continuity of the real world light signals has been lost; movementvector setting means for setting movement vectors of an object withinthe image data corresponding to the continuity of the real world lightsignals of which a part of the continuity has been lost in the imagedata; model generating means for modeling the region between pixelvalues of each of the pixels within the processing region and pixelvalues of each of the pixels where there is no movement blurring, withthe understanding that the pixel values of each of the pixels within theprocessing region are values obtained by pixel values of each of thepixels without movement blurring as to the object being integrated whilemoving corresponding to a movement vector; normal equation generatingmeans for generating normal equations from a first equation wherein thepixel values of the pixels within the processing region have beensubstituted into a model generated by the model generating means, and asecond equation for constraining the relation between the pixels wherethere is no movement blurring; and actual world estimating means forestimating the pixel values of each of the pixels where there is nomovement blurring, by computing the normal equations generated from thenormal equation generating means.

The normal equation generating means allow a normal equation to begenerated from a first equation wherein the pixel values of each of thepixels within the processing region are substituted as to the modelgenerated by the model generating means, and a second equation whereinthe difference of the pixel value between each of the pixels notgenerating movement blurring is zero.

A signal processing method according to the present invention comprises:a processing region setting step for setting the processing regionwithin data, wherein light signals of the real world have been projectedon a plurality of pixels each having time integration effects and a partof the continuity of the real world light signals has been lost; amovement vector setting step for setting movement vectors of an objectwithin the image data corresponding to the continuity of the real worldlight signals of which a part of the continuity has been lost in theimage data; a model generating step for modeling the region betweenpixel values of each of the pixels within the processing region andpixel values of each of the pixels where there is no movement blurring,with the understanding that the pixel values of each of the pixelswithin the processing region are values obtained by pixel values of eachof the pixels without movement blurring as to the object beingintegrated while moving corresponding to a movement vector; a normalequation generating step for generating normal equations from a firstequation wherein the pixel values of the pixels within the processingregion have been substituted into a model generated in the modelgenerating step, and a second equation for constraining the relationbetween the pixels where there is no movement blurring; and an actualworld estimating step for estimating the pixel values of each of thepixels where there is no movement blurring, by computing the normalequations generated from the normal equation generating step.

A program for a recording medium according to the present inventioncauses a computer to execute: a processing region setting step forsetting the processing region within data, wherein light signals of thereal world have been projected on a plurality of pixels each having timeintegration effects and a part of the continuity of the real world lightsignals has been lost; a movement vector setting step for settingmovement vectors of an object within the image data corresponding to thecontinuity of the real world light signals of which a part of thecontinuity has been lost in the image data; a model generating step formodeling the region between pixel values of each of the pixels withinthe processing region and pixel values of each of the pixels where thereis no movement blurring, with the understanding that the pixel values ofeach of the pixels within the processing region are values obtained bypixel values of each of the pixels without movement blurring as to theobject being integrated while moving corresponding to a movement vector;a normal equation generating step for generating normal equations from afirst equation wherein the pixel values of the pixels within theprocessing region have been substituted into a model generated in themodel generating step, and a second equation for constraining therelation between the pixels where there is no movement blurring; and anactual world estimating step for estimating the pixel values of each ofthe pixels where there is no movement blurring, by computing the normalequations generated from the normal equation generating step.

A program according to the present invention causes a computer toexecute: a processing region setting step for setting the processingregion within data, wherein light signals of the real world have beenprojected on a plurality of pixels each having time integration effectsand a part of the continuity of the real world light signals has beenlost; a movement vector setting step for setting movement vectors of anobject within the image data corresponding to the continuity of the realworld light signals of which a part of the continuity has been lost inthe image data; a model generating step for modeling the region betweenpixel values of each of the pixels within the processing region andpixel values of each of the pixels where there is no movement blurring,with the understanding that the pixel values of each of the pixelswithin the processing region are values obtained by pixel values of eachof the pixels without movement blurring as to the object beingintegrated while moving corresponding to a movement vector; a normalequation generating step for generating normal equations from a firstequation wherein the pixel values of the pixels within the processingregion have been substituted into a model generated in the modelgenerating step, and a second equation for constraining the relationbetween the pixels where there is no movement blurring; and an actualworld estimating step for estimating the pixel values of each of thepixels where there is no movement blurring, by computing the normalequations generated from the normal equation generating step.

With the present invention, a light signal of the real world isprojected on multiple pixels, each of which has a time integrationeffect, and assuming a processing region is set within the image datawherein a portion of the continuity of the light signal of the realworld has been lost, and a movement vector of the object within theimage data corresponding to the continuity of the light signal of thereal world wherein a portion of the continuity of the image data islost, is set, and the pixel values of each of the pixels within theprocessing region are said to be a value integrated while the pixelvalues of each of the pixels not generating movement blurringcorresponding to the object is shifted corresponding to the movementvector, and the relation between the pixel values of each of the pixelsof the processing region and the pixel values of each of the pixels notgenerating movement blurring is modeled, and a normal equation isgenerated as to the model, from the first equation wherein the pixelvalues of each of the pixels within the processing region aresubstituted and the second equation which constrains the relationbetween each of the pixels not generating movement blurring, and bycomputing the generated normal equation, pixel values for each of thepixels not generating movement blurring can be estimated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a block diagram illustrating an example of the hardwareconfiguration of a signal processing device 4.

FIG. 3 is a block diagram illustrating a configuration example of anembodiment of the signal processing device 4 shown in FIG. 1.

FIG. 4 is a diagram describing in detail the principle of the signalprocessing performed by the signal processing device 4.

FIG. 5 is a diagram for describing an example of array of pixels on animage sensor.

FIG. 6 is a diagram for describing the operations of a detecting devicewhich is a CCD.

FIG. 7 is a diagram for describing the relation between light cast intodetecting elements corresponding to pixel D through pixel F, and pixelvalues.

FIG. 8 is a diagram describing the relation between elapsing of time,light cast into a detecting element corresponding to one pixel, andpixel values.

FIG. 9 is a diagram illustrating an example of an image of a linearobject in the actual world 1.

FIG. 10 is a diagram illustrating an example of pixel values of imagedata obtained by actual imaging.

FIG. 11 is a diagram illustrating an example of an image of the actualworld 1, of an object which is of a color different from that of thebackground, having a monotone and linear edge.

FIG. 12 is a diagram illustrating an example of pixel values of imagedata obtained by actual imaging.

FIG. 13 is a schematic diagram of image data.

FIG. 14 is a diagram for describing estimation of a model 161 with Mpieces of data 162.

FIG. 15 is a diagram for describing the relation between signals of theactual world 1 and data 3.

FIG. 16 is a diagram illustrating an example of data 3 of interest atthe time of creating an Expression.

FIG. 17 is a diagram for describing signals for two objects in theactual world 1, and values belonging to a mixed region at the time ofcreating an expression.

FIG. 18 is a diagram for describing continuity represented by Expression(18), Expression (19), and Expression (22).

FIG. 19 is a diagram illustrating an example of M pieces of data 162extracted from data 3.

FIG. 20 is a diagram for describing integration of signals of the actualworld 1 in the time direction and two-dimensional spatial direction, inthe data 3.

FIG. 21 is a diagram for describing an integration region at the time ofgenerating high-resolution data with higher resolution in the spatialdirection.

FIG. 22 is a diagram for describing an integration region at the time ofgenerating high-resolution data with higher resolution in the timedirection.

FIG. 23 is a diagram for describing an integration region at the time ofgenerating high-resolution data with higher resolution in thetime-spatial directions.

FIG. 24 is a diagram illustrating the original image of the input image.

FIG. 25 is a diagram illustrating an example of an input image.

FIG. 26 is a diagram illustrating an image obtained by applyingconventional class classification adaptation processing.

FIG. 27 is a diagram illustrating results of detecting a fine lineregion.

FIG. 28 is a diagram illustrating an example of an output image outputfrom the signal processing device 4.

FIG. 29 is a flowchart for describing signal processing with the signalprocessing device 4.

FIG. 30 is a block diagram illustrating the configuration of a datacontinuity detecting unit 101.

FIG. 31 is a diagram illustrating an image in the actual world 1 with afine line in front of the background.

FIG. 32 is a diagram for describing approximation of a background with aplane.

FIG. 33 is a diagram illustrating the cross-sectional shape of imagedata regarding which the image of a fine line has been projected.

FIG. 34 is a diagram illustrating the cross-sectional shape of imagedata regarding which the image of a fine line has been projected.

FIG. 35 is a diagram illustrating the cross-sectional shape of imagedata regarding which the image of a fine line has been projected.

FIG. 36 is a diagram for describing the processing for detecting a peakand detecting of monotonous increase/decrease regions.

FIG. 37 is a diagram for describing the processing for detecting a fineline region wherein the pixel value of the peak exceeds a threshold,while the pixel value of the adjacent pixel is equal to or below thethreshold value.

FIG. 38 is a diagram representing the pixel value of pixels arrayed inthe direction indicated by dotted line AA′ in FIG. 37.

FIG. 39 is a diagram for describing processing for detecting continuityin a monotonous increase/decrease region.

FIG. 40 is a diagram illustrating an example of other processing fordetecting regions where an image of a fine line has been projected.

FIG. 41 is a flowchart for describing continuity detection processing.

FIG. 42 is a diagram for describing processing for detecting continuityof data in the time direction.

FIG. 43 is a block diagram illustrating the configuration of anon-continuity component extracting unit 201.

FIG. 44 is a diagram for describing the number of time of rejections.

FIG. 45 is a flowchart describing the processing for extracting thenon-continuity component.

FIG. 46 is a flowchart describing the processing for extracting thecontinuity component.

FIG. 47 is a flowchart describing other processing for extracting thecontinuity component.

FIG. 48 is a flowchart describing still other processing for extractingthe continuity component.

FIG. 49 is a block diagram illustrating another configuration of acontinuity component extracting unit 101.

FIG. 50 is a diagram for describing the activity in an input imagehaving data continuity.

FIG. 51 is a diagram for describing a block for detecting activity.

FIG. 52 is a diagram for describing the angle of data continuity as toactivity.

FIG. 53 is a block diagram illustrating a detailed configuration of thedata continuity detecting unit 101.

FIG. 54 is a diagram describing a set of pixels.

FIG. 55 is a diagram describing the relation between the position of apixel set and the angle of data continuity.

FIG. 56 is a flowchart for describing processing for detecting datacontinuity.

FIG. 57 is a diagram illustrating a set of pixels extracted whendetecting the angle of data continuity in the time direction and spacedirection.

FIG. 58 is a diagram for describing the principle of functionapproximation, which is an example of an embodiment of the actual worldestimating unit shown in FIG. 3.

FIG. 59 is a diagram for describing integration effects in the eventthat the sensor is a CCD.

FIG. 60 is a diagram for describing a specific example of theintegration effects of the sensor shown in FIG. 59.

FIG. 61 is a diagram for describing another specific example of theintegration effects of the sensor shown in FIG. 59.

FIG. 62 is a diagram representing a fine-line-inclusive actual worldregion shown in FIG. 60.

FIG. 63 is a diagram for describing the principle of an example of anembodiment of the actual world estimating unit shown in FIG. 3, incomparison with the example shown in FIG. 58.

FIG. 64 is a diagram representing the fine-line-inclusive data regionshown in FIG. 60.

FIG. 65 is a diagram wherein each of the pixel values contained in thefine-line-inclusive data region shown in FIG. 64 are plotted on a graph.

FIG. 66 is a diagram wherein an approximation function, approximatingthe pixel values contained in the fine-line-inclusive data region shownin FIG. 65, is plotted on a graph.

FIG. 67 is a diagram for describing the continuity in the spatialdirection which the fine-line-inclusive actual world region shown inFIG. 60 has.

FIG. 68 is a diagram wherein each of the pixel values contained in thefine-line-inclusive data region shown in FIG. 64 are plotted on a graph.

FIG. 69 is a diagram for describing a state wherein each of the inputpixel values indicated in FIG. 68 are shifted by a predetermined shiftamount.

FIG. 70 is a diagram wherein an approximation function, approximatingthe pixel values contained in the fine-line-inclusive data region shownin FIG. 65, is plotted on a graph, taking into consideration thespatial-direction continuity.

FIG. 71 is a diagram for describing space-mixed region.

FIG. 72 is a diagram for describing an approximation functionapproximating actual-world signals in a space-mixed region.

FIG. 73 is a diagram wherein an approximation function, approximatingthe actual world signals corresponding to the fine-line-inclusive dataregion shown in FIG. 65, is plotted on a graph, taking intoconsideration both the sensor integration properties and thespatial-direction continuity.

FIG. 74 is a block diagram for describing a configuration example of theactual world estimating unit using, of function approximation techniqueshaving the principle shown in FIG. 58, primary polynomial approximation.

FIG. 75 is a flowchart for describing actual world estimation processingwhich the actual world estimating unit of the configuration shown inFIG. 74 executes.

FIG. 76 is a diagram for describing a tap range.

FIG. 77 is a diagram for describing actual world signals havingcontinuity in the spatial direction.

FIG. 78 is a diagram for describing integration effects in the eventthat the sensor is a CCD.

FIG. 79 is a diagram for describing distance in the cross-sectionaldirection.

FIG. 80 is a block diagram for describing a configuration example of theactual world estimating unit using, of function approximation techniqueshaving the principle shown in FIG. 58, quadratic polynomialapproximation.

FIG. 81 is a flowchart for describing actual world estimation processingwhich the actual world estimating unit of the configuration shown inFIG. 80 executes.

FIG. 82 is a diagram for describing a tap range.

FIG. 83 is a diagram for describing direction of continuity in thetime-spatial direction.

FIG. 84 is a diagram for describing integration effects in the eventthat the sensor is a CCD.

FIG. 85 is a diagram for describing actual world signals havingcontinuity in the spatial direction.

FIG. 86 is a diagram for describing actual world signals havingcontinuity in the space-time directions.

FIG. 87 is a block diagram for describing a configuration example of theactual world estimating unit using, of function approximation techniqueshaving the principle shown in FIG. 58, cubic polynomial approximation.

FIG. 88 is a flowchart for describing actual world estimation processingwhich the actual world estimating unit of the configuration shown inFIG. 87 executes.

FIG. 89 is a diagram for describing the principle of re-integration,which is an example of an embodiment of the image generating unit shownin FIG. 3.

FIG. 90 is a diagram for describing an example of an input pixel and anapproximation function for approximation of an actual world signalcorresponding to the input pixel.

FIG. 91 is a diagram for describing an example of creating fourhigh-resolution pixels in the one input pixel shown in FIG. 90, from theapproximation function shown in FIG. 90.

FIG. 92 is a block diagram for describing a configuration example of animage generating unit using, of re-integration techniques having theprinciple shown in FIG. 89, one-dimensional re-integration technique.

FIG. 93 is a flowchart for describing the image generating processingwhich the image generating unit of the configuration shown in FIG. 92executes.

FIG. 94 is a diagram illustrating an example of the original image ofthe input image.

FIG. 95 is a diagram illustrating an example of image data correspondingto the image shown in FIG. 94.

FIG. 96 is a diagram representing an example of an input image.

FIG. 97 is a diagram representing an example of image data correspondingto the image shown in FIG. 96.

FIG. 98 is a diagram illustrating an example of an image obtained bysubjecting an input image to conventional class classificationadaptation processing.

FIG. 99 is a diagram representing an example of image data correspondingto the image shown in FIG. 98.

FIG. 100 is a diagram illustrating an example of an image obtained bysubjecting an input image to the one-dimensional re-integrationtechnique.

FIG. 101 is a diagram illustrating an example of image datacorresponding to the image shown in FIG. 100.

FIG. 102 is a diagram for describing actual-world signals havingcontinuity in the spatial direction.

FIG. 103 is a block diagram for describing a configuration example of animage generating unit which uses, of the re-integration techniqueshaving the principle shown in FIG. 89, a two-dimensional re-integrationtechnique.

FIG. 104 is a diagram for describing distance in the cross-sectionaldirection.

FIG. 105 is a flowchart for describing the image generating processingwhich the image generating unit of the configuration shown in FIG. 103executes.

FIG. 106 is a diagram for describing an example of an input pixel.

FIG. 107 is a diagram for describing an example of creating fourhigh-resolution pixels in the one input pixel shown in FIG. 106, withthe two-dimensional re-integration technique.

FIG. 108 is a diagram for describing the direction of continuity in thespace-time directions.

FIG. 109 is a block diagram for describing a configuration example ofthe image generating unit which uses, of the re-integration techniqueshaving the principle shown in FIG. 89, a three-dimensionalre-integration technique.

FIG. 110 is a flowchart for describing the image generating processingwhich the image generating unit of the configuration shown in FIG. 109executes.

FIG. 111 is a block diagram illustrating a configuration example ofanother embodiment of the signal processing device 4 shown in FIG. 1.

FIG. 112 is a flowchart for describing the processing of the signalprocessing device 4 shown in FIG. 111.

FIG. 113 is a block diagram illustrating a configuration example of anapplied embodiment of the signal processing device 4 shown in FIG. 111.

FIG. 114 is a flowchart for describing the processing of the signalprocessing device 4 shown in FIG. 113.

FIG. 115 is a diagram describing a light signal of an actual world 1.

FIG. 116 is a diagram describing the integration effect in the eventthat the sensor 2 is a CCD.

FIG. 117 is a diagram showing an approximation function f(x,y) of theactual world 1.

FIG. 118 is a diagram describing an input image to be input in thesignal processing device 4 of FIG. 113, and a processing region.

FIG. 119 is a diagram describing an example of a processing region setin the signal processing device 4 of the FIG. 113.

FIG. 120 is a diagram illustrating an approximation function f(x) of theX cross-section, wherein the value of y within the approximationfunction f(x,y) in FIG. 120 is a predetermined value y_(c)(y=y_(c)).

FIG. 121 is a diagram showing the approximation function f(x) in FIG.120 after 1/v time has passed.

FIG. 122 is a diagram showing the approximation function f(x) in FIG.121 after 1/v time has passed.

FIG. 123 is a diagram describing the movement of the object to be imagedin the input image moves v pixels for every shutter time.

FIG. 124 is a diagram showing the approximation function f(x) of y=y_(c)at a predetermined precise timing, illustrated in FIG. 120, with a X-tplane.

FIG. 125 is a diagram showing pixel values P₀ through P₉ generatingmovement blurring, with aX-t plane, so as to correspond with the pixelvalues Q₀ through Q₉ in FIG. 124.

FIG. 126 is a diagram showing the pixel values P₀ through P₉ of an inputimage with the pixel values Q₀ through Q₉ in FIG. 124.

FIG. 127 is a diagram describing a method for solving the blank regionsin FIG. 126, hypothesizing the edge portions of the processing region as“being flat”.

FIG. 128 is a diagram describing the line of interest of the processingregion in FIG. 119.

FIG. 129 is a block diagram illustrating a configuration example of theactual world estimating unit 15013 in FIG. 113.

FIG. 130 is a flowchart describing the actual world estimatingprocessing of step S15008 in FIG. 114.

FIG. 131 is a flowchart describing the actual world estimatingprocessing of step S15008 in FIG. 114.

FIG. 132 is a diagram illustrating the input image to be input into thesignal processing device 4 in FIG. 113.

FIG. 133 is a diagram illustrating a processed and output image whereinthe signal processing device 4 in FIG. 113 has processed the input imagein FIG. 132 with the first method, which adds an equation which ispostulated as having the edge portions of the processing regions as“being flat”.

FIG. 134 is a diagram illustrating a processed and output image whereinthe signal processing device 4 in FIG. 113 has processed the input imagein FIG. 132 with the second method, which adds a conditional expressionconstraining the relation between the adjacent pixel values.

FIG. 135 is a block diagram of a configuration example of anotherembodiment of an application example of the signal processing device 4in FIG. 111.

FIG. 136 is a flowchart describing the processing of the signalprocessing device 4 in FIG. 135.

FIG. 137 is a block diagram illustrating a configuration example of theactual world estimating unit 15083 in FIG. 135.

FIG. 138 is a flowchart describing the actual world estimatingprocessing in step S15088 in FIG. 136.

FIG. 139 is a flowchart describing the actual world estimatingprocessing in step S15088 in FIG. 136.

FIG. 140 is a block diagram of a configuration example of anotherembodiment of an application example of the signal processing device 4in FIG. 111.

FIG. 141 is a flowchart describing the processing of the signalprocessing device 4 in FIG. 140.

FIG. 142 is a block diagram illustrating a configuration example of theactual world estimating unit 15113 in FIG. 140.

FIG. 143 is a flowchart describing the actual world estimatingprocessing in step S15168 in FIG. 141.

FIG. 144 is a diagram illustrating the input image which is input intothe signal processing device 4 in FIG. 140.

FIG. 145 is a diagram illustrating an example of an output image whereinthe signal processing device 4 in FIG. 135 processes the input image inFIG. 144 using the weighting W_(bj) corresponding to the constraintcondition expression as the same weighting for all of the constraintcondition expressions.

FIG. 146 is a diagram illustrating an example of an output image whereinthe signal processing device 4 in FIG. 140 processes the input image inFIG. 144 using the weighting W_(bj) corresponding to the constraintcondition expression as a 0 or a 1, according to the activity of theinput image.

FIG. 147 is a block diagram of a configuration example of anotherembodiment of an application example of the signal processing device 4in FIG. 111.

FIG. 148 is a flowchart describing the processing of the signalprocessing device 4 in FIG. 147.

FIG. 149 is a block diagram illustrating a configuration example of theactual world estimating unit 15153 in FIG. 147.

FIG. 150 is a flowchart for describing the actual world estimatingprocessing in step S15218 in FIG. 148.

FIG. 151 is a flowchart for describing the actual world estimatingprocessing in step S15218 in FIG. 148.

FIG. 152 is a block diagram illustrating a configuration example of acontinuity setting unit 15012 shown in FIG. 113.

FIG. 153 is a diagram for describing amount of movement.

FIG. 154 is a diagram illustrating pixel values of an image output froma camera, taken by the camera while a foreground object passes in frontof a background object.

FIG. 155 is a diagram illustrating difference values of the pixel valuesof pixels in the image shown in FIG. 154.

FIG. 156 is a flowchart for describing processing for detecting amountof movement.

FIG. 157 is a flowchart for describing processing for detectingcorrelation.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 illustrates the principle of the present invention. As shown inthe drawing, events (phenomena) in an actual world 1 having dimensionsof space, time, and mass, are acquired by a sensor 2, and formed intodata. Events in the actual world 1 refer to light (images), sound,pressure, temperature, mass, humidity, brightness/darkness, or smells,and so forth. The events in the actual world 1 are distributed in thespace-time directions. For example, an image of the actual world 1 is adistribution of the intensity of light of the actual world 1 in thespace-time directions.

Taking note of the sensor 2, of the events in the actual world 1 havingthe dimensions of space, time, and mass, the events in the actual world1 which the sensor 2 can acquire, are converted into data 3 by thesensor 2. It can be said that information indicating events in theactual world 1 are acquired by the sensor 2.

That is to say, the sensor 2 converts information indicating events inthe actual world 1, into data 3. It can be said that signals which areinformation indicating the events (phenomena) in the actual world 1having dimensions of space, time, and mass, are acquired by the sensor 2and formed into data.

Hereafter, the distribution of events such as images, sound, pressure,temperature, mass, humidity, brightness/darkness, or smells, and soforth, in the actual world 1, will be referred to as signals of theactual world 1, which are information indicating events. Also, signalswhich are information indicating events of the actual world 1 will alsobe referred to simply as signals of the actual world 1. In the presentSpecification, signals are to be understood to include phenomena andevents, and also include those wherein there is no intent on thetransmitting side.

The data 3 (detected signals) output from the sensor 2 is informationobtained by projecting the information indicating the events of theactual world 1 on a space-time having a lower dimension than the actualworld 1. For example, the data 3 which is image data of a moving image,is information obtained by projecting an image of the three-dimensionalspace direction and time direction of the actual world 1 on thetime-space having the two-dimensional space direction and timedirection. Also, in the event that the data 3 is digital data forexample, the data 3 is rounded off according to the sampling increments.In the event that the data 3 is analog data, information of the data 3is either compressed according to the dynamic range, or a part of theinformation has been deleted by a limiter or the like.

Thus, by projecting the signals shown are information indicating eventsin the actual world 1 having a predetermined number of dimensions ontodata 3 (detection signals), a part of the information indicating eventsin the actual world 1 is lost. That is to say, a part of the informationindicating events in the actual world 1 is lost from the data 3 whichthe sensor 2 outputs.

However, even though a part of the information indicating events in theactual world 1 is lost due to projection, the data 3 includes usefulinformation for estimating the signals which are information indicatingevents (phenomena) in the actual world 1.

With the present invention, information having continuity contained inthe actual world 1 or the data 3 is used as useful information forestimating the signals which is information of the actual world 1.Continuity is a concept which is newly defined.

Taking note of the actual world 1, events in the actual world 1 includecharacteristics which are constant in predetermined dimensionaldirections. For example, an object (corporeal object) in the actualworld 1 either has shape, pattern, or color that is continuous in thespace direction or time direction, or has repeated patterns of shape,pattern, or color.

Accordingly, the information indicating the events in actual world 1includes characteristics constant in a predetermined dimensionaldirection.

With a more specific example, a linear object such as a string, cord, orrope, has a characteristic which is constant in the length-wisedirection, i.e., the spatial direction, that the cross-sectional shapeis the same at arbitrary positions in the length-wise direction. Thecontinuity in the spatial direction that the cross-sectional shape isthe same at arbitrary positions in the length-wise direction comes fromthe characteristic that the linear object is long.

Accordingly, an image of the linear object has a characteristic which isconstant in the length-wise direction, i.e., the spatial direction, thatthe cross-sectional shape is the same, at arbitrary positions in thelength-wise direction.

Also, a monotone object, which is a corporeal object, having an expansein the spatial direction, can be said to have a continuity of having thesame color in the spatial direction regardless of the part thereof.

In the same way, an image of a monotone object, which is a corporealobject, having an expanse in the spatial direction, have a continuity ofhaving the same color in the spatial direction regardless of the partthereof.

In this way, events in the actual world 1 (real world) havecharacteristics which are constant in predetermined dimensionaldirections, so signals of the actual world 1 have characteristics whichare constant in predetermined dimensional directions

In the present Specification, such characteristics which are constant inpredetermined dimensional directions will be called continuity.Continuity of the signals of the actual world 1 (real world) means thecharacteristics which are constant in predetermined dimensionaldirections which the signals indicating the events of the actual world 1(real world) have.

Countless such continuities exist in the actual world 1 (real world).

Next, taking note of the data 3, the data 3 is obtained by signals whichis information indicating events of the actual world 1 havingpredetermined dimensions being projected by the sensor 2, and includescontinuity corresponding to the continuity of signals in the real world.It can be said that the data 3 includes continuity wherein thecontinuity of actual world signals has been projected.

However, as described above, in the data 3 output from the sensor 2, apart of the information of the actual world 1 has been lost, so a partof the continuity contained in the signals of the actual world 1 (realworld) may be lost from the data.

In other words, the data 3 contains at least a part of the continuitywithin the continuity of the signals of the actual world 1 (real world)as data continuity. Data continuity means characteristics which areconstant in predetermined dimensional directions, which the data 3 has.

With the present invention, continuity of the actual world 1 signals, orthe data continuity which the data 3 has, is used as significant datafor estimating signals which are information indicating events of theactual world 1.

For example, with the signal processing device 4, information indicatingan event in the actual world 1 which has been lost is generated bysignals processing of the data 3, using data continuity.

Now, with the signal processing device 4, of the length (space), time,and mass, which are dimensions of signals serving as informationindicating events in the actual world 1, continuity in the spatialdirection or time direction, are used.

In FIG. 1, the sensor 2 is formed of, for example, a digital stillcamera, a video camera, or the like, and takes images of the actualworld 1, and outputs the image data which is the obtained data 3, to asignal processing device 4. The sensor 2 may also be a thermographydevice, a pressure sensor using photo-elasticity, or the like.

The signal processing device 4 is configured of, for example, a personalcomputer or the like, and performs signal processing with regard to thedata 3.

The signal processing device 4 is configured as shown in FIG. 2, forexample. A CPU (Central Processing Unit) 21 executes various types ofprocessing following programs stored in ROM (Read Only Memory) 22 or thestorage unit 28. RAM (Random Access Memory) 23 stores programs to beexecuted by the CPU 21, data, and so forth, as suitable. The CPU 21, ROM22, and RAM 23, are mutually connected by a bus 24.

Also connected to the CPU 21 is an input/output interface 25 via the bus24. An input device 26 made up of a keyboard, mouse, microphone, and soforth, and an output unit 27 made up of a display, speaker, and soforth, are connected to the input/output interface 25. The CPU 21executes various types of processing corresponding to commands inputfrom the input unit 26. The CPU 21 then outputs images and audio and thelike obtained as a result of processing to the output unit 27.

A storage unit 28 connected to the input/output interface 25 isconfigured of a hard disk for example, and stores the programs andvarious types of data which the CPU 21 executes. A communication unit 29communicates with external devices via the Internet and other networks.In the case of this example, the communication unit 29 acts as anacquiring unit for capturing data 3 output from the sensor 2.

Also, an arrangement may be made wherein programs are obtained via thecommunication unit 29 and stored in the storage unit 28.

A drive 30 connected to the input/output interface 25 drives a magneticdisk 51, optical disk 52, magneto-optical disk 53, or semiconductormemory 54 or the like mounted thereto, and obtains programs and datarecorded therein. The obtained programs and data are transferred to thestorage unit 28 as necessary and stored.

FIG. 3 is a block diagram illustrating a signal processing device 4.

Note that whether the functions of the signal processing device 4 arerealized by hardware or realized by software is irrelevant. That is tosay, the block diagrams in the present Specification may be taken to behardware block diagrams or may be taken to be software function blockdiagrams.

With the signal processing device 4 of which the configuration is shownin FIG. 3, image data which is an example of the data 3 is input, andthe continuity of the data is detected from the input image data (inputimage). Next, the signals of the actual world 1 acquired by the sensor 2are estimated from the continuity of the data detected. Then, based onthe estimated signals of the actual world 1, an image is generated, andthe generated image (output image) is output. That is to say, FIG. 3 isa diagram illustrating the configuration of the signal processing device4 which is an image processing device.

The input image (image data which is an example of the data 3) input tothe signal processing device 4 is supplied to a data continuitydetecting unit 101 and actual world estimating unit 102.

The data continuity detecting unit 101 detects the continuity of thedata from the input image, and supplies data continuity informationindicating the detected continuity to the actual world estimating unit102 and an image generating unit 103. The data continuity informationincludes, for example, the position of a region of pixels havingcontinuity of data, the direction of a region of pixels havingcontinuity of data (the angle or gradient of the time direction andspace direction), or the length of a region of pixels having continuityof data, or the like in the input image. Detailed configuration of thedata continuity detecting unit 101 will be described later. The actualworld estimating unit 102 estimates the signals of the actual world 1,based on the input image and the data continuity information suppliedfrom the data continuity detecting unit 101. That is to say, the actualworld estimating unit 102 estimates an image which is the signals of theactual world cast into the sensor 2 at the time that the input image wasacquired. The actual world estimating unit 102 supplies the actual worldestimation information indicating the results of the estimation of thesignals of the actual world 1, to the image generating unit 103. Thedetailed configuration of the actual world estimating unit 102 will bedescribed later.

The image generating unit 103 generates signals further approximatingthe signals of the actual world 1, based on the actual world estimationinformation indicating the estimated signals of the actual world 1,supplied from the actual world estimating unit 102, and outputs thegenerated signals. Or, the image generating unit 103 generates signalsfurther approximating the signals of the actual world 1, based on thedata continuity information supplied from the data continuity detectingunit 101, and the actual world estimation information indicating theestimated signals of the actual world 1, supplied from the actual worldestimating unit 102, and outputs the generated signals.

That is to say, the image generating unit 103 generates an image furtherapproximating the image of the actual world 1 based on the actual worldestimation information, and outputs the generated image as an outputimage. Or, the image generating unit 103 generates an image furtherapproximating the image of the actual world 1 based on the datacontinuity information and actual world estimation information, andoutputs the generated image as an output image.

For example, the image generating unit 103 generates an image withhigher resolution in the spatial direction or time direction incomparison with the input image, by integrating the estimated image ofthe actual world 1 within a desired range of the spatial direction ortime direction, based on the actual world estimation information, andoutputs the generated image as an output image. For example, the imagegenerating unit 103 generates an image by extrapolation/interpolation,and outputs the generated image as an output image.

Detailed configuration of the image generating unit 103 will bedescribed later.

Next, the principle of the present invention will be described withreference to FIG. 4.

For example, signals of the actual world 1, which are an image forexample, are imaged on the photoreception face of a CCD (Charge CoupledDevice) which is an example of the sensor 2. The CCD, which is anexample of the sensor 2, has integration properties, so difference isgenerated in the data 3 output from the CCD as to the image of theactual world 1. Details of the integration properties of the sensor 2will be described later.

With the signal processing by the signal processing device 4, therelation between the image of the actual world 1 obtained by the CCD,and the data 3 taken by the CCD and output, is explicitly taken intoconsideration. That is to say, the relation between the data 3 and thesignals which is information of the actual world obtained by the sensor2, is explicitly taken into consideration.

More specifically, as shown in FIG. 4, the signal processing device 4uses a model 161 to approximate (describe) the actual world 1. The model161 is represented by, for example, N variables. More accurately, themodel 161 approximates (describes) signals of the actual world 1.

In order to predict the model 161, the signal processing device 4extracts M pieces of data 162 from the data 3. At the time of extractingthe M pieces of data 162 from the data 3, the signal processing device 4uses the continuity of the data contained in the data 3, for example. Inother words, the signal processing device 4 extracts data 162 forpredicting the model 161, based on the continuity of the data containedin the data 3. Consequently, in this case, the model 161 is constrainedby the continuity of the data.

That is to say, the model 161 approximates (information (signals)indicating) events of the actual world 1 having continuity (continuityin a predetermined dimensional direction), which generates the datacontinuity in the data 3 when acquired with the sensor 2.

Now, in the event that the number M of the data 162 is N or more, whichis the number of variables of the model, the model 161 represented bythe N variables can be predicted, from the M pieces of the data 162.

In this way, the signal processing device 4 can take into considerationthe signals which are information of the actual world 1, by predictingthe model 161 approximating (describing) the (signals of the) actualworld 1.

Next, the integration effects of the sensor 2 will be described.

An image sensor such as a CCD or CMOS (Complementary Metal-OxideSemiconductor) sensor or the like, which is the sensor 2 for takingimages, projects signals, which are information of the real world, ontotwo-dimensional data, at the time of imaging the real world. The pixelsof the image sensor each have a predetermined area, as a so-calledphotoreception face (photoreception region). Incident light to thephotoreception face having a predetermined area is integrated in thespace direction and time direction for each pixel, and is converted intoa single pixel value for each pixel.

The space-time integration of images will be described with reference toFIG. 5 through FIG. 8.

An image sensor images a subject (object) in the real world, and outputsthe obtained image data as a result of imagining in increments of singleframes. That is to say, the image sensor acquires signals of the actualworld 1 which is light reflected off of the subject of the actual world1, and outputs the data 3.

For example, the image sensor outputs image data of 30 frames persecond. In this case, the exposure time of the image sensor can be madeto be 1/30 seconds. The exposure time is the time from the image sensorstarting conversion of incident light into electric charge, to ending ofthe conversion of incident light into electric charge. Hereafter, theexposure time will also be called shutter time.

FIG. 5 is a diagram describing an example of a pixel array on the imagesensor. In FIG. 5, A through I denote individual pixels. The pixels areplaced on a plane corresponding to the image displayed by the imagedata. A single detecting element corresponding to a single pixel isplaced on the image sensor. At the time of the image sensor takingimages of the actual world 1, the one detecting element outputs onepixel value corresponding to the one pixel making up the image data. Forexample, the position in the spatial direction X (X coordinate) of thedetecting element corresponds to the horizontal position on the imagedisplayed by the image data, and the position in the spatial direction Y(Y coordinate) of the detecting element corresponds to the verticalposition on the image displayed by the image data.

Distribution of intensity of light of the actual world 1 has expanse inthe three-dimensional spatial directions and the time direction, but theimage sensor acquires light of the actual world 1 in two-dimensionalspatial directions and the time direction, and generates data 3representing the distribution of intensity of light in thetwo-dimensional spatial directions and the time direction.

As shown in FIG. 6, the detecting device which is a CCD for example,converts light cast onto the photoreception face (photoreception region)(detecting region) into electric charge during a period corresponding tothe shutter time, and accumulates the converted charge. The light isinformation (signals) of the actual world 1 regarding which theintensity is determined by the three-dimensional spatial position andpoint-in-time. The distribution of intensity of light of the actualworld 1 can be represented by a three-dimensional function F(x, y, z,t), wherein position x, y, z, in space, and point-in-time t, arevariables.

The amount of charge accumulated in the detecting device which is a CCDis approximately proportionate to the intensity of the light cast ontothe entire photoreception face having two-dimensional spatial expanse,and the amount of time that light is cast thereupon. The detectingdevice adds the charge converted from the light cast onto the entirephotoreception face, to the charge already accumulated during a periodcorresponding to the shutter time. That is to say, the detecting deviceintegrates the light cast onto the entire photoreception face having atwo-dimensional spatial expanse, and accumulates a change of an amountcorresponding to the integrated light during a period corresponding tothe shutter time. The detecting device can also be said to have anintegration effect regarding space (photoreception face) and time(shutter time).

The charge accumulated in the detecting device is converted into avoltage value by an unshown circuit, the voltage value is furtherconverted into a pixel value such as digital data or the like, and isoutput as data 3. Accordingly, the individual pixel values output fromthe image sensor have a value projected on one-dimensional space, whichis the result of integrating the portion of the information (signals) ofthe actual world 1 having time-space expanse with regard to the timedirection of the shutter time and the spatial direction of thephotoreception face of the detecting device.

That is to say, the pixel value of one pixel is represented as theintegration of F(x, y, t). F(x, y, t) is a function representing thedistribution of light intensity on the photoreception face of thedetecting device. For example, the pixel value P is represented byExpression (1).

In Expression (1), x₁ represents the spatial coordinate at the left-sideboundary of the photoreception face of the detecting device (Xcoordinate). x₂ represents the spatial coordinate at the right-sideboundary of the photoreception face of the detecting device (Xcoordinate). In Expression (1), y₁ represents the spatial coordinate atthe top-side boundary of the photoreception face of the detecting device(Y coordinate). y₂ represents the spatial coordinate at the bottom-sideboundary of the photoreception face of the detecting device (Ycoordinate). Also, t₁ represents the point-in-time at which conversionof incident light into an electric charge was started. t₂ represents thepoint-in-time at which conversion of incident light into an electriccharge was ended.

Note that actually, the gain of the pixel values of the image dataoutput from the image sensor is corrected for the overall frame, forexample.

Each of the pixel values of the image data are integration values of thelight cast on the photoreception face of each of the detecting elementsof the image sensor, and of the light cast onto the image sensor,waveforms of light of the actual world 1 finer than the photoreceptionface of the detecting element are hidden in the pixel value asintegrated values.

Hereafter, in the present Specification, the waveform of signalsrepresented with a predetermined dimension as a reference may bereferred to simply as waveforms.

Thus, the image of the actual world 1 (light signals) is integrated inthe spatial direction and time direction in increments of pixels, so apart of the continuity of the image of the actual world 1 drops out fromthe image data, so another part of the continuity of the image of theactual world 1 is left in the image data. Or, there may be cases whereincontinuity which has changed from the continuity of the image of theactual world 1 is included in the image data.

Further description will be made regarding the integration effect in thespatial direction for an image taken by an image sensor havingintegration effects.

FIG. 7 is a diagram describing the relation between incident light tothe detecting elements corresponding to the pixel D through pixel F, andthe pixel values. F(x) in FIG. 7 is an example of a functionrepresenting the distribution of light intensity of the actual world 1,having the coordinate x in the spatial direction X in space (on thedetecting device) as a variable. In other words, F(x) is an example of afunction representing the distribution of light intensity of the actualworld 1, with the spatial direction Y and time direction constant. InFIG. 7, L indicates the length in the spatial direction X of thephotoreception face of the detecting device corresponding to the pixel Dthrough pixel F.

The pixel value of a single pixel is represented as the integral ofF(x). For example, the pixel value P of the pixel E is represented byExpression (2).

In Expression (2), x₁ represents the spatial coordinate in the spatialdirection X at the left-side boundary of the photoreception face of thedetecting device corresponding to the pixel E. x₂ represents the spatialcoordinate in the spatial direction X at the right-side boundary of thephotoreception face of the detecting device corresponding to the pixelE.

In the same way, further description will be made regarding theintegration effect in the time direction for an image taken by an imagesensor having integration effects.

FIG. 8 is a diagram for describing the relation between time elapsed,the incident light to a detecting element corresponding to a singlepixel, and the pixel value. F(t) in FIG. 8 is a function representingthe distribution of light intensity of the actual world 1, having thepoint-in-time t as a variable. In other words, F(t) is an example of afunction representing the distribution of light intensity of the actualworld 1, with the spatial direction Y and the spatial direction Xconstant. T_(s) represents the shutter time.

The frame #n−1 is a frame which is previous to the frame #n time-wise,and the frame #n+1 is a frame following the frame #n time-wise. That isto say, the frame #n−1, frame #n, and frame #n+1, are displayed in theorder of frame #n−1, frame #n, and frame #n+1.

Note that in the example shown in FIG. 8, the shutter time t_(s) and theframe intervals are the same.

The pixel value of a single pixel is represented as the integral ofF(t). For example, the pixel value P of the pixel of frame #n isrepresented by Expression (3).

In Expression (3), t₁ represents the time at which conversion ofincident light into an electric charge was started. t₂ represents thetime at which conversion of incident light into an electric charge wasended.

Hereafter, the integration effect in the spatial direction by the sensor2 will be referred to simply as spatial integration effect, and theintegration effect in the time direction by the sensor 2 also will bereferred to simply as time integration effect. Also, space integrationeffects or time integration effects will be simply called integrationeffects.

Next, description will be made regarding an example of continuity ofdata included in the data 3 acquired by the image sensor havingintegration effects.

FIG. 9 is a diagram illustrating a linear object of the actual world 1(e.g., a fine line), i.e., an example of distribution of lightintensity. In FIG. 9, the position to the upper side of the drawingindicates the intensity (level) of light, the position to the upperright side of the drawing indicates the position in the spatialdirection X which is one direction of the spatial directions of theimage, and the position to the right side of the drawing indicates theposition in the spatial direction Y which is the other direction of thespatial directions of the image.

The image of the linear object of the actual world 1 includespredetermined continuity. That is to say, the image shown in FIG. 9 hascontinuity in that the cross-sectional shape (the change in level as tothe change in position in the direction orthogonal to the lengthdirection), at any arbitrary position in the length direction.

FIG. 10 is a diagram illustrating an example of pixel values of imagedata obtained by actual image-taking, corresponding to the image shownin FIG. 9.

That is to say, FIG. 10 is a model diagram of the image data obtained byimaging, with the image sensor, an image of a linear object having adiameter shorter than the length L of the photoreception face of eachpixel, and extending in a direction offset from the array of the pixelsof the image sensor (the vertical or horizontal array of the pixels).The image cast into the image sensor at the time that the image datashown in FIG. 10 was acquired is an image of the linear object of theactual world 1 shown in FIG. 9.

In FIG. 10, the position to the upper side of the drawing indicates thepixel value, the position to the upper right side of the drawingindicates the position in the spatial direction X which is one directionof the spatial directions of the image, and the position to the rightside of the drawing indicates the position in the spatial direction Ywhich is the other direction of the spatial directions of the image. Thedirection indicating the pixel value in FIG. 10 corresponds to thedirection of level in FIG. 9, and the spatial direction X and spatialdirection Y in FIG. 10 also are the same as the directions in FIG. 9.

In the event of taking an image of a linear object having a diametershorter than the length L of the photoreception face of each pixel withthe image sensor, the linear object is represented in the image dataobtained as a result of the image-taking as multiple arc shapes(half-discs) having a predetermined length which are arrayed in adiagonally-offset fashion, in a model representation, for example. Thearc shapes are of approximately the same shape. One arc shape is formedon one row of pixels vertically, or is formed on one row of pixelshorizontally. For example, one arc shape shown in FIG. 10 is formed onone row of pixels vertically.

Thus, with the image data taken and obtained by the image sensor forexample, the continuity in that the cross-sectional shape in the spatialdirection Y at any arbitrary position in the length direction is thesame, which the linear object image of the actual world 1 had, is lost.Also, it can be said that the continuity, which the linear object imageof the actual world 1 had, has changed into continuity in that arcshapes of the same shape formed on one row of pixels vertically orformed on one row of pixels horizontally are arrayed at predeterminedintervals.

FIG. 11 is a diagram illustrating an image in the actual world 1 of anobject having a straight edge, and is of a monotone color different fromthat of the background, i.e., an example of distribution of lightintensity. In FIG. 11, the position to the upper side of the drawingindicates the intensity (level) of light, the position to the upperright side of the drawing indicates the position in the spatialdirection X which is one direction of the spatial directions of theimage, and the position to the right side of the drawing indicates theposition in the spatial direction Y which is the other direction of thespatial directions of the image.

The image of the object of the actual world 1 which has a straight edgeand is of a monotone color different from that of the background,includes predetermined continuity. That is to say, the image shown inFIG. 11 has continuity in that the cross-sectional shape (the change inlevel as to the change in position in the direction orthogonal to theedge) is the same at any arbitrary position in the length direction ofthe edge.

FIG. 12 is a diagram illustrating an example of pixel values of theimage data obtained by actual image-taking, corresponding to the imageshown in FIG. 11. As shown in FIG. 12, the image data is in a steppedshape, since the image data is made up of pixel values in increments ofpixels.

FIG. 13 is a model diagram illustrating the image data shown in FIG. 12.

The model diagram shown in FIG. 13 is a model diagram of image dataobtained by taking, with the image sensor, an image of the object of theactual world 1 which has a straight edge and is of a monotone colordifferent from that of the background, and extending in a directionoffset from the array of the pixels of the image sensor (the vertical orhorizontal array of the pixels). The image cast into the image sensor atthe time that the image data shown in FIG. 13 was acquired is an imageof the object of the actual world 1 which has a straight edge and is ofa monotone color different from that of the background, shown in FIG.11.

In FIG. 13, the position to the upper side of the drawing indicates thepixel value, the position to the upper right side of the drawingindicates the position in the spatial direction X which is one directionof the spatial directions of the image, and the position to the rightside of the drawing indicates the position in the spatial direction Ywhich is the other direction of the spatial directions of the image. Thedirection indicating the pixel value in FIG. 13 corresponds to thedirection of level in FIG. 11, and the spatial direction X and spatialdirection Y in FIG. 13 also are the same as the directions in FIG. 11.

In the event of taking an image of an object of the actual world 1 whichhas a straight edge and is of a monotone color different from that ofthe background with an image sensor, the straight edge is represented inthe image data obtained as a result of the image-taking as multiple pawlshapes having a predetermined length which are arrayed in adiagonally-offset fashion, in a model representation, for example. Thepawl shapes are of approximately the same shape. One pawl shape isformed on one row of pixels vertically, or is formed on one row ofpixels horizontally. For example, one pawl shape shown in FIG. 13 isformed on one row of pixels vertically.

Thus, the continuity of image of the object of the actual world 1 whichhas a straight edge and is of a monotone color different from that ofthe background, in that the cross-sectional shape is the same at anyarbitrary position in the length direction of the edge, for example, islost in the image data obtained by imaging with an image sensor. Also,it can be said that the continuity, which the image of the object of theactual world 1 which has a straight edge and is of a monotone colordifferent from that of the background had, has changed into continuityin that pawl shapes of the same shape formed on one row of pixelsvertically or formed on one row of pixels horizontally are arrayed atpredetermined intervals.

The data continuity detecting unit 101 detects such data continuity ofthe data 3 which is an input image, for example. For example, the datacontinuity detecting unit 101 detects data continuity by detectingregions having a continuity in a predetermined dimensional direction.For example, the data continuity detecting unit 101 detects a regionwherein the same arc shapes are arrayed at constant intervals, such asshown in FIG. 10. Also, for example, the data continuity detecting unit101 detects a region wherein the same pawl shapes are arrayed atconstant intervals, such as shown in FIG. 13.

Also, the data continuity detecting unit 101 detects continuity of thedata by detecting angle (gradient) in the spatial direction, indicatingan array of the same shapes.

Also, for example, the data continuity detecting unit 101 detectscontinuity of data by detecting angle (movement) in the space directionand time direction, indicating the array of the same shapes in the spacedirection and the time direction.

Further, for example, the data continuity detecting unit 101 detectscontinuity in the data by detecting the length of the region havingcontinuity in a predetermined dimensional direction.

Hereafter, the portion of data 3 where the sensor 2 has projected theimage of the object of the actual world 1 which has a straight edge andis of a monotone color different from that of the background, will alsobe called a two-valued edge.

Now, with conventional signal processing, desired high-resolution data,for example, is generated from the data 3.

Conversely, with the signal processing by the signal processing device4, the actual world 1 is estimated from the data 3, and thehigh-resolution data is generated based on the estimation results. Thatis to say, the actual world 1 is estimated from the data 3, and thehigh-resolution data is generated based on the estimated actual world 1,taking into consideration the data 3.

In order to generate the high-resolution data from the actual world 1,there is the need to take into consideration the relation between theactual world 1 and the data 3. For example, how the actual world 1 isprojected on the data 3 by the sensor 2 which is a CCD, is taken intoconsideration.

The sensor 2 which is a CCD has integration properties as describedabove. That is to say, one unit of the data 3 (e.g., pixel value) can becalculated by integrating a signal of the actual world 1 with adetection region (e.g., photoreception face) of a detection device(e.g., CCD) of the sensor 2.

Applying this to the high-resolution data, the high-resolution data canbe obtained by applying processing, wherein a virtual high-resolutionsensor projects signals of the actual world 1 to the data 3, to theestimated actual world 1.

In other words, if the signals of the actual world 1 can be estimatedfrom the data 3, one value contained in the high-resolution data can beobtained by integrating signals of the actual world 1 for each detectionregion of the detecting elements of the virtual high-resolution sensor(in the time-space direction).

For example, in the event that the change in signals of the actual world1 are smaller than the size of the detection region of the detectingelements of the sensor 2, the data 3 cannot expresses the small changesin the signals of the actual world 1. Accordingly, high-resolution dataindicating small change of the signals of the actual world 1 can beobtained by integrating the signals of the actual world 1 estimated fromthe data 3 with each region (in the time-space direction) that issmaller in comparison with the change in signals of the actual world 1.

That is to say, integrating the signals of the estimated actual world 1with the detection region with regard to each detecting element of thevirtual high-resolution sensor enables the high-resolution data to beobtained.

With the signal processing device 4, the image generating unit 103generates the high-resolution data by integrating the signals of theestimated actual world 1 in the time-space direction regions of thedetecting elements of the virtual high-resolution sensor, for example.

Next, in order to estimate the actual world 1 from the data 3, at thesignal processing device 4, the relationship between the data 3 and theactual world 1, continuity, and a spatial or temporal mixture in thedata 3 (space mixture or time mixture), are used.

Here, a mixture means a value in the data 3 wherein the signals of twoobjects in the actual world 1 are mixed to yield a single value.

A space mixture means the mixture of the signals of two objects in thespatial direction due to the spatial integration effects of the sensor2. Time mixture will be described later.

The actual world 1 itself is made up of countless events, andaccordingly, in order to represent the actual world 1 itself withmathematical expressions, for example, there is the need to have aninfinite number of variables. It is impossible to predict all events ofthe actual world 1 from the data 3.

In the same way, it is impossible to predict all of the signals of theactual world 1 from the data 3.

Accordingly, with the signal processing device 4, of the signals of theactual world 1, a portion which has continuity and which can beexpressed by the function f(x, y, z, t) is taken note of, and theportion of the signals of the actual world 1 which can be represented bythe function f(x, y, z, t) and has continuity is approximated with amodel 161 represented by N variables. As shown in FIG. 14, the model 161is predicted from the M pieces of data 162 in the data 3.

In order to enable the model 161 to be predicted from the M pieces ofdata 162, first, there is the need to represent the model 161 with Nvariables based on the continuity, and second, to generate an expressionusing the N variables which indicates the relation between the model 161represented by the N variables and the M pieces of data 162 based on theintegral properties of the sensor 2. Since the model 161 is representedby the N variables, based on the continuity, it can be said that theexpression using the N variables that indicates the relation between themodel 161 represented by the N variables and the M pieces of data 162,describes the relation between the part of the signals of the actualworld 1 having continuity, and the part of the data 3 having datacontinuity.

In other words, the part of the signals of the actual world 1 havingcontinuity, that is approximated by the model 161 represented by the Nvariables, generates data continuity in the data 3.

The data continuity detecting unit 101 detects the part of the data 3where data continuity has been generated by the part of the signals ofthe actual world 1 having continuity, and the characteristics of thepart where data continuity has been generated.

For example, as shown in FIG. 15, in an image of the object of theactual world 1 which has a straight edge and is of a monotone colordifferent from that of the background, the edge at the position ofinterest indicated by A in FIG. 15, has a gradient. The arrow B in FIG.15 indicates the gradient of the edge. A predetermined edge gradient canbe represented as an angle as to a reference axis or as a direction asto a reference position. For example, a predetermined edge gradient canbe represented as the angle between the coordinates axis of the spatialdirection X and the edge. For example, the predetermined edge gradientcan be represented as the direction indicated by the length of thespatial direction X and the length of the spatial direction Y.

At the time that the image of the object of the actual world 1 which hasa straight edge and is of a monotone color different from that of thebackground is obtained at the sensor 2 and the data 3 is output, pawlshapes corresponding to the edge are arrayed in the data 3 at theposition corresponding to the position of interest (A) of the edge inthe image of the actual world 1, which is indicated by A′ in FIG. 15,and pawl shapes corresponding to the edge are arrayed in the directioncorresponding to the gradient of the edge of the image in the actualworld 1, in the direction of the gradient indicated by B′ in FIG. 15.

The model 161 represented with the N variables approximates such aportion of the signals of the actual world 1 generating data continuityin the data 3.

At the time of formulating an expression using the N variablesindicating the relation between the model 161 represented with the Nvariables and the M pieces of data 162, the values of part where datacontinuity is generated in the data 3 are used.

In this case, in the data 3 shown in FIG. 16, taking note of the valueswhere data continuity is generated and which belong to a mixed region,an expression is formulated with a value integrating the signals of theactual world 1 as being equal to a value output by the detecting elementof the sensor 2. For example, multiple expressions can be formulatedregarding the multiple values in the data 3 where data continuity isgenerated.

In FIG. 16, A denotes the position of interest of the edge, and A′denotes (the position of) the pixel corresponding to the position (A) ofinterest of the edge in the image of the actual world 1.

Now, a mixed region means a region of data in the data 3 wherein thesignals for two objects in the actual world 1 are mixed and become onevalue. For example, a pixel value wherein, in the image of the object ofthe actual world 1 which has a straight edge and is of a monotone colordifferent from that of the background in the data 3, the image of theobject having the straight edge and the image of the background areintegrated, belongs to a mixed region.

FIG. 17 is a diagram describing signals for two objects in the actualworld 1 and values belonging to a mixed region, in a case of formulatingan expression.

FIG. 17 illustrates, to the left, signals of the actual world 1corresponding to two objects in the actual world 1 having apredetermined expansion in the spatial direction X and the spatialdirection Y. which are acquired at the detection region of a singledetecting element of the sensor 2. FIG. 17 illustrates, to the right, apixel value P of a single pixel in the data 3 wherein the signals of theactual world 1 illustrated to the left in FIG. 17 have been projected bya single detecting element of the sensor 2. That is to say, illustratesa pixel value P of a single pixel in the data 3 wherein the signals ofthe actual world 1 corresponding to two objects in the actual world 1having a predetermined expansion in the spatial direction X and thespatial direction Y which are acquired by a single detecting element ofthe sensor 2, have been projected.

L in FIG. 17 represents the level of the signal of the actual world 1which is shown in white in FIG. 17, corresponding to one object in theactual world 1. R in FIG. 17 represents the level of the signal of theactual world 1 which is shown hatched in FIG. 17, corresponding to theother object in the actual world 1.

Here, the mixture ratio α is the ratio of (the area of) the signalscorresponding to the two objects cast into the detecting region of theone detecting element of the sensor 2 having a predetermined expansionin the spatial direction X and the spatial direction Y. For example, themixture ratio α represents the ratio of area of the level L signals castinto the detecting region of the one detecting element of the sensor 2having a predetermined expansion in the spatial direction X and thespatial direction Y, as to the area of the detecting region of a singledetecting element of the sensor 2.

In this case, the relation between the level L, level R, and the pixelvalue P, can be represented by Expression (4).α×L+f(1−α)×R=P  (4)Note that there may be cases wherein the level R may be taken as thepixel value of the pixel in the data 3 positioned to the right side ofthe pixel of interest, and there may be cases wherein the level L may betaken as the pixel value of the pixel in the data 3 positioned to theleft side of the pixel of interest.

Also, the time direction can be taken into consideration in the same wayas with the spatial direction for the mixture ratio α and the mixedregion. For example, in the event that an object in the actual world 1which is the object of image-taking, is moving as to the sensor 2, theratio of signals for the two objects cast into the detecting region ofthe single detecting element of the sensor 2 changes in the timedirection. The signals for the two objects regarding which the ratiochanges in the time direction, that have been cast into the detectingregion of the single detecting element of the sensor 2, are projectedinto a single value of the data 3 by the detecting element of the sensor2.

The mixture of signals for two objects in the time direction due to timeintegration effects of the sensor 2 will be called time mixture.

The data continuity detecting unit 101 detects regions of pixels in thedata 3 where signals of the actual world 1 for two objects in the actualworld 1, for example, have been projected. The data continuity detectingunit 101 detects gradient in the data 3 corresponding to the gradient ofan edge of an image in the actual world 1, for example.

The actual world estimating unit 102 estimates the signals of the actualworld 1 by formulating an expression using N variables, representing therelation between the model 161 represented by the N variables and the Mpieces of data 162, based on the region of the pixels having apredetermined mixture ratio α detected by the data continuity detectingunit 101 and the gradient of the region, for example, and solving theformulated expression.

Description will be made further regarding specific estimation of theactual world 1.

Of the signals of the actual world represented by the function F(x, y,z, t)let us consider approximating the signals of the actual worldrepresented by the function F(x, y, t) at the cross-section in thespatial direction Z (the position of the sensor 2), with anapproximation function f(x, y, t) determined by a position x in thespatial direction X, a position y in the spatial direction Y, and apoint-in-time t.

Now, the detection region of the sensor 2 has an expanse in the spatialdirection X and the spatial direction Y. In other words, theapproximation function f(x, y, t) is a function approximating thesignals of the actual world 1 having an expanse in the spatial directionand time direction, which are acquired with the sensor 2.

Let us say that projection of the signals of the actual world 1 from thesensor 2 yields a value P(x, y, t) of the data 3. The value P(x, y, t)of the data 3 is a pixel value which the sensor 2 which is an imagesensor outputs, for example.

Now, in the event that the projection by the sensor 2 can be formulated,the value obtained by projecting the approximation function f(x, y, t)can be represented as a projection function S(x, y, t).

Obtaining the projection function S(x, y, t) has the following problems.

First, generally, the function F(x, y, z, t) representing the signals ofthe actual world 1 can be a function with an infinite number of orders.

Second, even if the signals of the actual world could be described as afunction, the projection function S(x, y, t) via projection of thesensor 2 generally cannot be determined. That is to say, the action ofprojection by the sensor 2, in other words, the relation between theinput signals and output signals of the sensor 2, is unknown, so theprojection function S(x, y, t) cannot be determined.

With regard to the first problem, let us consider expressing thefunction f(x, y, t) approximating signals of the actual world 1 with thesum of products of the function f_(i)(x, y, t) which is a describablefunction (e.g., a function with a finite number of orders) and variablesw_(i).

Also, with regard to the second problem, formulating projection by thesensor 2 allows us to describe the function S_(i)(x, y, t) from thedescription of the function f_(i)(x, y, t).

That is to say, representing the function f(x, y, t) approximatingsignals of the actual world 1 with the sum of products of the functionf_(i)(x, y, t) and variables w_(i), Expression (5) can be obtained.

For example, as indicated in Expression (6), the relation between thedata 3 and the signals of the actual world can be formulated as shown inExpression (7) from Expression (5) by formulating the projection of thesensor 2.

In Expression (7), j represents the index of the data.

In the event that M data groups (j=1 through M) common with the Nvariables w_(i) (i=1 through N) exists in Expression (7), Expression (8)is satisfied, so the model 161 of the actual world can be obtained fromdata 3.N≦M  (8)

N is the number of variables representing the model 161 approximatingthe actual world 1. M is the number of pieces of data 162 include in thedata 3.

Representing the function f(x, y, t) approximating the actual world 1signals with Expression (5) allows the variable portion w_(i) to behandled independently. At this time, i represents the number ofvariables. Also, the form of the function represented by f_(i) can behanded independently, and a desired function can be used for f_(i).

Accordingly, the number N of the variables w_(i) can be defined withoutdependence on the function f_(i), and the variables w_(i) can beobtained from the relation between the number N of the variables w_(i)and the number of pieces of data M.

That is to say, using the following three allows the actual world 1 tobe estimated from the data 3.

First, the N variables are determined. That is to say, Expression (5) isdetermined. This enables describing the actual world 1 using continuity.For example, the signals of the actual world 1 can be described with amodel 161 wherein a cross-section is expressed with a polynomial, andthe same cross-sectional shape continues in a constant direction.

Second, for example, projection by the sensor 2 is formulated,describing Expression (7). For example, this is formulated such that theresults of integration of the signals of the actual world 2 are data 3.

Third, M pieces of data 162 are collected to satisfy Expression (8). Forexample, the data 162 is collected from a region having data continuitythat has been detected with the data continuity detecting unit 101. Forexample, data 162 of a region wherein a constant cross-sectioncontinues, which is an example of continuity, is collected.

In this way, the relation between the data 3 and the actual world 1 isdescribed with Expression (5), and M pieces of data 162 are collected,thereby satisfying Expression (8), and the actual world 1 can beestimated.

More specifically, in the event of N=M, the number of variables N andthe number of expressions M are equal, so the variables w_(i) can beobtained by formulating a simultaneous equation.

Also, in the event that N<M, various solving methods can be applied. Forexample, the variables w_(i) can be obtained by least-square.

Now, the solving method by least-square will be described in detail

First, an Expression (9) for predicting data 3 from the actual world 1will be shown according to Expression (7).

In Expression (9), P′_(j)(x_(j), y_(j), t_(j)) is a prediction value.

The sum of squared differences E for the prediction value P′ andobserved value P is represented by Expression (10).

The variables w_(i) are obtained such that the sum of squareddifferences E is the smallest. Accordingly, the partial differentialvalue of Expression (10) for each variable W_(k) is 0. That is to say,Expression (11) holds.

Expression (11) yields Expression (12).

When Expression (12) holds with K=1 through N, the solution byleast-square is obtained. The normal equation thereof is shown inExpression (13).

Note that in Expression (13), S_(i)(x_(j), y_(j), t_(j)) is described asS_(i)(j)

From Expression (14) through Expression (16), Expression (13) can beexpressed as S_(MAT)W_(MAT)=P_(MAT).

In Expression (13), S_(i) represents the projection of the actual world1. In Expression (13), P_(j) represents the data 3. In Expression (13),w_(i) represents variables for describing and obtaining thecharacteristics of the signals of the actual world 1.

Accordingly, inputting the data 3 into Expression (13) and obtainingW_(MAT) by a matrix solution or the like enables the actual world 1 tobe estimated. That is to say, the actual world 1 can be estimated bycomputing Expression (17).W _(MAT) =S _(MAT) ⁻¹ P _(MAT)  (17)

Note that in the event that S_(MAT) is not regular, a transposed matrixof S_(MAT) can be used to obtain W_(MAT).

The actual world estimating unit 102 estimates the actual world 1 by,for example, inputting the data 3 into Expression (13) and obtainingW_(MAT) by a matrix solution or the like.

Now, an even more detailed example will be described. For example, thecross-sectional shape of the signals of the actual world 1, i.e., thechange in level as to the change in position, will be described with apolynomial. Let us assume that the cross-sectional shape of the signalsof the actual world 1 is constant, and that the cross-section of thesignals of the actual world 1 moves at a constant speed. Projection ofthe signals of the actual world 1 from the sensor 2 to the data 3 isformulated by three-dimensional integration in the time-space directionof the signals of the actual world 1.

The assumption that the cross-section of the signals of the actual world1 moves at a constant speed yields Expression (18) and Expression (19).

Here, v_(x) and v_(y) are constant.

Using Expression (18) and Expression (19), the cross-sectional shape ofthe signals of the actual world 1 can be represented as in Expression(20).

$\begin{matrix}{{f\left( {x^{\prime},y^{\prime}} \right)} = {f\left( {{x + {v_{x}t}},{y + {v_{y}t}}} \right)}} & (20)\end{matrix}$

Formulating projection of the signals of the actual world 1 from thesensor 2 to the data 3 by three-dimensional integration in thetime-space direction of the signals of the actual world 1 yieldsExpression (21).

In Expression (21), S(x, y, t) represents an integrated value the regionfrom position x_(s) to position x_(e) for the spatial direction X, fromposition y_(s) to position y_(e) for the spatial direction Y, and frompoint-in-time t_(s) to point-in-time t_(e) for the time direction t,i.e., the region represented as a space-time cuboid.

Solving Expression (13) using a desired function f(x′, y′) wherebyExpression (21) can be determined enables the signals of the actualworld 1 to be estimated.

In the following, we will use the function indicated in Expression (22)as an example of the function f(x′, y′).

That is to say, the signals of the actual world 1 are estimated toinclude the continuity represented in Expression (18), Expression (19),and Expression (22). This indicates that the cross-section with aconstant shape is moving in the space-time direction as shown in FIG.18.

Substituting Expression (22) into Expression (21) yields Expression(23).

wherein

Volume=(x_(e)−x_(s)) (y_(e)−y_(s)) (t_(e)−t_(s))

S₀(x, y, t)=Volume/2×(x_(e)+x_(s)+v_(x)(t_(e)+t_(s)))

S₁(x, y, t)=Volume/2×(y_(e)+y_(s)+v_(y)(t_(e)+t_(s)))

S₂(x, y, t)=1 holds.

FIG. 19 is a diagram illustrating an example of the M pieces of data 162extracted from the data 3. For example, let us say that 27 pixel valuesare extracted as the data 162, and that the extracted pixel values areP_(j)(x, y, t) In this case, j is 0 through 26.

In the example shown in FIG. 19, in the event that the pixel value ofthe pixel corresponding to the position of interest at the point-in-timet which is n is P₁₃(x, y, t), and the direction of array of the pixelvalues of the pixels having the continuity of data (e.g., the directionin which the same-shaped pawl shapes detected by the data continuitydetecting unit 101 are arrayed) is a direction connecting P₄(X, y, t),P₁₃(x, y, t), and P₂₂(x, y, t), the pixel values P₉(x, y, t) throughP₁₇(x, y, t) at the point-in-time t which is n, the pixel values P₀(x,y, t) through P₈(x, y, t) at the point-in-time t which is n−1 which isearlier in time than n, and the pixel values P₁₈(x, y, t) through P₂₆(x,y, t) at the point-in-time t which is n+1 which is later in time than n,are extracted.

Now, the region regarding which the pixel values, which are the data 3output from the image sensor which is the sensor 2, have been obtained,have a time-direction and two-dimensional spatial direction expansion.Now, for example, the center of gravity of the cuboid corresponding tothe pixel values (the region regarding which the pixel values have beenobtained) can be used as the position of the pixel in the space-timedirection.

Generating Expression (13) from the 27 pixel values P₀(x, y, t) throughP₂₆(x, y, t) and from Expression (23), and obtaining W, enables theactual world 1 to be estimated.

In this way, the actual world estimating unit 102 generates Expression(13) from the 27 pixel values P₀(x, y, t) through P₂₆(x, y, t) and fromExpression (23), and obtains W, thereby estimating the signals of theactual world 1.

Note that a Gaussian function, a sigmoid function, or the like, can beused for the function f_(i)(x, y, t).

An example of processing for generating high-resolution data with evenhigher resolution, corresponding to the data 3, from the estimatedactual world 1 signals, will be described with reference to FIG. 20through FIG. 23.

As shown in FIG. 20, the data 3 has a value wherein signals of theactual world 1 are integrated in the time direction and two-dimensionalspatial directions. For example, a pixel value which is data 3 that hasbeen output from the image sensor which is the sensor 2 has a valuewherein the signals of the actual world 1, which is light cast into thedetecting device, are integrated by the shutter time which is thedetection time in the time direction, and integrated by thephotoreception region of the detecting element in the spatial direction.

Conversely, as shown in FIG. 21, the high-resolution data with evenhigher resolution in the spatial direction is generated by integratingthe estimated actual world 1 signals in the time direction by the sametime as the detection time of the sensor 2 which has output the data 3,and also integrating in the spatial direction by a region narrower incomparison with the photoreception region of the detecting element ofthe sensor 2 which has output the data 3.

Note that at the time of generating the high-resolution data with evenhigher resolution in the spatial direction, the region where theestimated signals of the actual world 1 are integrated can be setcompletely disengaged from photoreception region of the detectingelement of the sensor 2 which has output the data 3. For example, thehigh-resolution data can be provided with resolution which is that ofthe data 3 magnified in the spatial direction by an integer, of course,and further, can be provided with resolution which is that of the data 3magnified in the spatial direction by a rational number such as 5/3times, for example.

Also, as shown in FIG. 22, the high-resolution data with even higherresolution in the time direction is generated by integrating theestimated actual world 1 signals in the spatial direction by the sameregion as the photoreception region of the detecting element of thesensor 2 which has output the data 3, and also integrating in the timedirection by a time shorter than the detection time of the sensor 2which has output the data 3.

Note that at the time of generating the high-resolution data with evenhigher resolution in the time direction, the time by which the estimatedsignals of the actual world 1 are integrated can be set completelydisengaged from shutter time of the detecting element of the sensor 2which has output the data 3. For example, the high-resolution data canbe provided with resolution which is that of the data 3 magnified in thetime direction by an integer, of course, and further, can be providedwith resolution which is that of the data 3 magnified in the timedirection by a rational number such as 7/4 times, for example.

High-resolution data with movement blurring removed is generated byintegrating the estimated actual world 1 signals only in the spatialdirection and not in the time direction.

Further, as shown in FIG. 23, high-resolution data with higherresolution in the time direction and space direction is generated byintegrating the estimated actual world 1 signals in the spatialdirection by a region narrower in comparison with the photoreceptionregion of the detecting element of the sensor 2 which has output thedata 3, and also integrating in the time direction by a time shorter incomparison with the detection time of the sensor 2 which has output thedata 3.

In this case, the region and time for integrating the estimated actualworld 1 signals can be set completely unrelated to the photoreceptionregion and shutter time of the detecting element of the sensor 2 whichhas output the data 3.

Thus, the image generating unit 103 generates data with higherresolution in the time direction or the spatial direction, byintegrating the estimated actual world 1 signals by a desired space-timeregion, for example.

Accordingly, data which is more accurate with regard to the signals ofthe actual world 1, and which has higher resolution in the timedirection or the space direction, can be generated by estimating thesignals of the actual world 1.

FIG. 24 through FIG. 28 illustrate an example of an input image and anexample of the results of processing with the signal processing device 4used for signal processing.

FIG. 24 is a diagram illustrating an original image of an input image(equivalent to light signals of the actual world 1). FIG. 25 is adiagram illustrating an example of an input image. The input image shownin FIG. 25 is an image generated by taking the average value of pixelvalues of pixels belonging to blocks made up of 2 by 2 pixels of theimage shown in FIG. 24, as the pixel value of a single pixel. That is tosay, the input image is an image obtained by applying spatial directionintegration to the image shown in FIG. 24, imitating the integratingproperties of the sensor.

The original image shown in FIG. 24 contains an image of a fine lineinclined at approximately 5 degrees in the clockwise direction from thevertical direction. In the same way, the input image shown in FIG. 25contains an image of a fine line inclined at approximately 5 degrees inthe clockwise direction from the vertical direction.

FIG. 26 is a diagram illustrating an image obtained by applyingconventional class classification adaptation processing to the inputimage shown in FIG. 25. Now, class classification processing is made upof class classification processing and adaptation processing, whereinthe data is classified based on the nature thereof by the classclassification adaptation processing, and subjected to adaptationprocessing for each class. In the adaptation processing, a low-imagequality or standard image quality image, for example, is converted intoa high image quality image by being subjected to mapping (mapping) usinga predetermined tap coefficient.

That is to say, with adaptation processing, first data is converted intosecond data by being mapped (mapping) using a predetermined tapcoefficient.

Now, adaptation processing will be described with regard to a mappingmethod using the tap coefficient, wherein for example, a linearcombination model is used, and also a low-resolution image or standardresolution SD (Standard Definition) image obtained by filtering ahigh-resolution HD (High Definition) image with a low-pass filter isused as the first data, and the HD image used for obtaining the SD imageis used as the second data.

Now, under the above-described conditions, an HD pixel y making up theHD image can be obtained with the following linear Expression (linearcombination) using multiple SD pixels extracted from SD pixels making upthe SD image, as a prediction tap for predicting the HD image, and thetap coefficient.

Wherein, in Expression (24), x_(n) represents the pixel value of then'th pixel in the SD image, making up a prediction tap regarding the HDpixel y, and w_(n) represents the n'th tap coefficient to be multipliedby (the pixel value of) the n'th SD pixel. Note that in Expression (24),a prediction tap is made up of an N number of SD pixels x₁, x₂, . . . ,x_(N).

Now, the pixel value y of the HD pixel can be obtained by a quadraticexpression or higher, instead of the linear expression shown inExpression (24).

Now, in the HD image, saying that y_(k) represents the true value of(the pixel value of) the k'th HD pixel, and y_(k)′ represents aprediction value of the true value y_(k) obtained by Expression (24),the prediction error e_(k) thereof is as expressed in the followingExpression, for example.e _(k) =y _(k) −y _(k)′  (25)

The prediction value y_(k)′ in Expression (25) is obtained according toExpression (24), so substituting the y_(k)′ in Expression (25) accordingExpression (24) yields the following Expression.

Wherein, in Expression (26), X_(n,k) represents the n'th SD pixel makingup the prediction tap for the k'th HD pixel.

While a tap coefficient w_(n) wherein the prediction error e_(k) is 0 inExpression (26) is optimal for predicting the HD pixel, obtaining such atap coefficient w_(n) for all HD pixels is generally difficult.

Accordingly, as a rule representing the tap coefficient w_(n) as beingoptimal, employing the least-square method for example, means that theoptimal tap coefficient w_(n) can be obtained by minimizing thesummation E of squared errors represented in the following Expressionfor example, as a statistical error.

Wherein, in Expression (27), K represents the number of samples of setsmade up of an HD pixel y_(k) and SD pixels X_(1,k), X_(2,k), . . . ,X_(N,k) making up a prediction tap regarding that HD pixel y_(k).

The tap coefficient W_(n) which makes the summation E of squared errorsin Expression (27) smallest (the minimum) is such that the partialdifferentiation of the summation E by the tap coefficient w_(n) yields0. Accordingly, the following Expression must be satisfied.

Now, partial differentiation of the above Expression (26) by the tapcoefficient w_(n) yields the following Expression.

Expressions (28) and (29) yield the following Expression.

Substituting Expression (26) for e_(k) in Expression (30) allowsExpression (30) to be expressed in the form of the normal equation inExpression (31).

Preparing a certain number of sets of the HD pixel y_(k) and SD pixelsx_(n,k) allows as many of the normal equations in Expression (31) to beformulated as the number of tap coefficients w_(n) to be obtained, andsolving Expression (31) allows optimal tap coefficients w_(n) to beobtained. Note that sweeping (Gauss-Jordan elimination) or the like, forexample, can be used for solving Expression (31).

As described above, adaptation processing involves taking a great numberof HD pixels y₁, y₂, . . . , y_(K) as tutor data to serve as a tutor forlearning for tap coefficients, and also taking SD pixels X_(1,k),X_(2,k), . . . , X_(N,k) making up a prediction tap regarding each HDpixel y_(k) as student data to serve as a student for learning for tapcoefficients, and solving Expression (31), thereby performing learningfor obtaining an optimal tap coefficient w_(n) and further using theoptimal tap coefficient w_(n) to perform mapping (conversion) of an SDimage into an HD image according to Expression (24).

Now, for the SD pixels X_(1,k), X_(2,k), . . . , X_(N,k) making up theprediction tap regarding the HD pixels Y_(k), an SD pixel close to aposition on the SD image corresponding to the HD pixel y_(k), eitherspatially or temporally, can be used.

Also, with class classification adaptation processing, learning of thetap coefficient w_(n) and mapping using the tap coefficient w_(n) areperformed by class. With class classification adaptation processing,class classification processing is performed with regard to the HD pixely_(k) of interest, and learning of the tap coefficient w_(n) and mappingusing the tap coefficient w_(n) are performed for each class obtained bythe class classification processing.

An example of class classification processing with regard to the HDpixel y_(k) is to extract multiple SD pixels from the SD image to serveas a class tap used for class classification of the HD pixel y_(k), andto perform M-bit ADRC (Adaptive Dynamic Range Coding) using the classtap made up of the multiple SD pixels.

In M-bit ADRC processing, the maximum value MAX and minimum value MIN ofthe SD pixels making up the class tap are detected, DR=MAX−MIN is set asa local dynamic range, and the SD pixels making up the class tap arere-quantized into K bits, based on this dynamic range DR. That is tosay, the minimum value MIN is subtracted from the SD pixels making upthe class tap, and the subtraction value is divided byDR/2^(K)(quantized). Accordingly, in the event that a class tap issubjected to 1-bit ADRC processing for example, each SD pixel making upthe class tap is one bit. In this case, a bit string obtained byarraying in a predetermined order the 1-bit pixel values regarding eachSD pixel making up the class tap that have been obtained as describedabove is output as ADRC code, and the ADRC code is taken as a class coderepresenting the class.

Note that class classification adaptation processing differs from simpleinterpolation processing or the like, for example, in that componentsnot included in the SD image but are included in the HD image arereproduced. That is to say, with class classification adaptationprocessing, it would seem by looking at Expression (24) alone that thisis the same as interpolation processing using a so-called interpolationfilter, but the tap coefficient w_(n) which is equivalent to the tapcoefficient of the interpolation filter has been obtained by learningusing an HD image serving as tutor data and an SD image serving asstudent data, so the component contained in the HD image can bereproduced.

Now, tap coefficients w_(n) which perform various types of conversioncan be obtained in the tap coefficient w_(n) learning, depending on whatsort of combination of tutor data y and student data x is employed.

That is to say, in a case of taking a high-resolution HD image as thetutor data y and taking an SD image wherein the resolution of the HDimage has been deteriorated as the student data x, for example, a tapcoefficient w_(n) for mapping an image into an image with the resolutionimproved, can be obtained. Further, in a case of taking an HD image asthe tutor data y and taking an SD image wherein the number of pixels ofthe HD image has been reduced as the student data x, for example, a tapcoefficient w_(n) for mapping an image into an image with an increasednumber of pixels making up the image, can be obtained.

FIG. 26 is an image obtained by subjecting the input image shown in FIG.25 to mapping by the class classification adaptation processing such asdescribed above. It can be understood in the image shown in FIG. 26 thatthe image of the fine line is different to that of the original image inFIG. 24.

FIG. 27 is a diagram illustrating the results of detecting the fine lineregions from the input image shown in the example in FIG. 25, by thedata continuity detecting unit 101. In FIG. 27, the white regionindicates the fine line region, i.e., the region wherein the arc shapesshown in FIG. 10 are arrayed.

FIG. 28 is a diagram illustrating an example of the output imageobtained by performing signal processing at the signal processing device4, with the image shown in FIG. 25 as the input image. As shown in FIG.28, the signals processing device 4 yields an image closer to the fineline image of the original image shown in FIG. 24.

FIG. 29 is a flowchart for describing the processing of signals with thesignal processing device 4.

In step S101, the data continuity detecting unit 101 executes theprocessing for detecting continuity. The data continuity detecting unit101 detects data continuity contained in the input image which is thedata 3, and supplies the data continuity information indicating thedetected data continuity to the actual world estimating unit 102 and theimage generating unit 103.

The data continuity detecting unit 101 detects the continuity of datacorresponding to the continuity of the signals of the actual world. Inthe processing in step S101, the continuity of data detected by the datacontinuity detecting unit 101 is either part of the continuity of theimage of the actual world 1 contained in the data 3, or continuity whichhas changed from the continuity of the signals of the actual world 1.

For example, the data continuity detecting unit 101 detects the datacontinuity by detecting a region having a continuity in a predetermineddimensional direction. Also, for example, the data continuity detectingunit 101 detects data continuity by detecting angle (gradient) in thespatial direction indicating the array of the same shape.

Details of the continuity detecting processing in step S101 will bedescribed later.

Note that the data continuity information can be used as features,indicating the characteristics of the data 3.

In step S102, the actual world estimating unit 102 executes processingfor estimating the actual world. That is to say, the actual worldestimating unit 102 estimates the signals of the actual world based onthe input image and the data continuity information supplied from thedata continuity detecting unit 101. In the processing in step S102 forexample, the actual world estimating unit 102 estimates the signals ofthe actual world 1 by predicting a model 161 approximating (describing)the actual world 1. The actual world estimating unit 102 supplies theactual world estimation information indicating the estimated signals ofthe actual world 1 to the image generating unit 103.

For example, the actual world estimating unit 102 estimates the actualworld 1 signals by predicting the width of the linear object. Also, forexample, the actual world estimating unit 102 estimates the actual world1 signals by predicting a level indicating the color of the linearobject.

Details of processing for estimating the actual world in step S102 willbe described later.

Note that the actual world estimation information can be used asfeatures, indicating the characteristics of the data 3.

In step S103, the image generating unit 103 performs image generatingprocessing, and the processing ends. That is to say, the imagegenerating unit 103 generates an image based on the actual worldestimation information, and outputs the generated image. Or, the imagegenerating unit 103 generates an image based on the data continuityinformation and actual world estimation information, and outputs thegenerated image.

For example, in the processing in step S103, the image generating unit103 integrates the estimated real world light in the spatial direction,based on the actual world estimated information, thereby generating animage with higher resolution in the spatial direction in comparison withthe input image, and outputs the generated image. For example, the imagegenerating unit 103 integrates estimated real world light in thetime-space direction, based on the actual world estimated information,hereby generating an image with higher resolution in the time directionand the spatial direction in comparison with the input image, andoutputs the generated image. The details of the image generatingprocessing in step S103 will be described later.

Thus, the signal processing device 4 detects data continuity from thedata 3, and estimates the actual world 1 from the detected datacontinuity. The signal processing device 4 then generates signals closerapproximating the actual world 1 based on the estimated actual world 1.

As described above, in the event of performing the processing forestimating signals of the real world, accurate and highly-preciseprocessing results can be obtained.

Also, in the event that first signals which are real world signalshaving first dimensions are projected, the continuity of datacorresponding to the lost continuity of the real world signals isdetected for second signals of second dimensions, having a number ofdimensions fewer than the first dimensions, from which a part of thecontinuity of the signals of the real world has been lost, and the firstsignals are estimated by estimating the lost real world signalscontinuity based on the detected data continuity, accurate andhighly-precise processing results can be obtained as to the events inthe real world.

Next, the details of the configuration of the data continuity detectingunit 101 will be described.

FIG. 30 is a block diagram illustrating the configuration of the datacontinuity detecting unit 101.

Upon taking an image of an object which is a fine line, the datacontinuity detecting unit 101, of which the configuration is shown inFIG. 30, detects the continuity of data contained in the data 3, whichis generated from the continuity in that the cross-sectional shape whichthe object has is the same. That is to say, the data continuitydetecting unit 101 of the configuration shown in FIG. 30 detects thecontinuity of data contained in the data 3, which is generated from thecontinuity in that the change in level of light as to the change inposition in the direction orthogonal to the length-wise direction is thesame at an arbitrary position in the length-wise direction, which theimage of the actual world 1 which is a fine line, has.

More specifically, the data continuity detecting unit 101 of whichconfiguration is shown in FIG. 30 detects the region where multiple arcshapes (half-disks) having a predetermined length are arrayed in adiagonally-offset adjacent manner, within the data 3 obtained by takingan image of a fine line with the sensor 2 having spatial integrationeffects.

The data continuity detecting unit 101 extracts the portions of theimage data (hereafter referred to as con-continuity component) otherthan the portion of the image data where the image of the fine linehaving data continuity has been projected (hereafter, the portion of theimage data where the image of the fine line having data continuity hasbeen projected will also be called continuity component, and the otherportions will be called non-continuity component), from an input imagewhich is the data 3, detects the pixels where the image of the fine lineof the actual world 1 has been projected, from the extractednon-continuity component and the input image, and detects the region ofthe input image made up of pixels where the image of the fine line ofthe actual world 1 has been projected.

A non-continuity component extracting unit 201 extracts thenon-continuity component from the input image, and supplies thenon-continuity component information indicating the extractednon-continuity component to a peak detecting unit 202 and a monotonousincrease/decrease detecting unit 203 along with the input image.

For example, as shown in FIG. 31, in the event that an image of theactual world 1 wherein a fine line exists in front of a background withan approximately constant light level is projected on the data 3, thenon-continuity component extracting unit 201 extracts the non-continuitycomponent which is the background, by approximating the background inthe input image which is the data 3, on a plane, as shown in FIG. 32. InFIG. 32, the solid line indicates the pixel values of the data 3, andthe dotted line illustrates the approximation values indicated by theplane approximating the background. In FIG. 32, A denotes the pixelvalue of the pixel where the image of the fine line has been projected,and the PL denotes the plane approximating the background.

In this way, the pixel values of the multiple pixels at the portion ofthe image data having data continuity are discontinuous as to thenon-continuity component.

The non-continuity component extracting unit 201 detects thediscontinuous portion of the pixel values of the multiple pixels of theimage data which is the data 3, where an image which is light signals ofthe actual world 1 has been projected and a part of the continuity ofthe image of the actual world 1 has been lost.

Details of the processing for extracting the non-continuity componentwith the non-continuity component extracting unit 201 will be describedlater.

The peak detecting unit 202 and the monotonous increase/decreasedetecting unit 203 remove the non-continuity component from the inputimage, based on the non-continuity component information supplied fromthe non-continuity component extracting unit 201. For example, the peakdetecting unit 202 and the monotonous increase/decrease detecting unit203 remove the non-continuity component from the input image by settingthe pixel values of the pixels of the input image where only thebackground image has been projected, to 0. Also, for example, the peakdetecting unit 202 and the monotonous increase/decrease detecting unit203 remove the non-continuity component from the input image bysubtracting values approximated by the plane PL from the pixel values ofeach pixel of the input image.

Since the background can be removed from the input image, the peakdetecting unit 202 through continuousness detecting unit 204 can processonly the portion of the image data where the fine line has be projected,thereby further simplifying the processing by the peak detecting unit202 through the continuousness detecting unit 204.

Note that the non-continuity component extracting unit 201 may supplyimage data wherein the non-continuity component has been removed fromthe input image, to the peak detecting unit 202 and the monotonousincrease/decrease detecting unit 203.

In the example of processing described below, the image data wherein thenon-continuity component has been removed from the input image, i.e.,image data made up from only pixel containing the continuity component,is the object.

Now, description will be made regarding the image data upon which thefine line image has been projected, which the peak detecting unit 202through continuousness detecting unit 204 are to detect.

In the event that there is no optical LPF, the cross-dimensional shapein the spatial direction Y (change in the pixel values as to change inthe position in the spatial direction) of the image data upon which thefine line image has been projected as shown in FIG. 31 can be thought tobe the trapezoid shown in FIG. 33, or the triangle shown in FIG. 34,from the spatial; integration effects of the image sensor which is thesensor 2. However, ordinary image sensors have an optical LPF with theimage sensor obtaining the image which has passed through the opticalLPF and projects the obtained image on the data 3, so in reality, thecross-dimensional shape of the image data with fine lines in the spatialdirection Y has a shape resembling Gaussian distribution, as shown inFIG. 35.

The peak detecting unit 202 through continuousness detecting unit 204detect a region made up of pixels upon which the fine line image hasbeen projected wherein the same cross-sectional shape (change in thepixel values as to change in the position in the spatial direction) isarrayed in the vertical direction in the screen at constant intervals,and further, detect a region made up of pixels upon which the fine lineimage has been projected which is a region having data continuity, bydetecting regional connection corresponding to the length-wise directionof the fine line of the actual world 1. That is to say, the peakdetecting unit 202 through continuousness detecting unit 204 detectregions wherein arc shapes (half-disc shapes) are formed on a singlevertical row of pixels in the input image, and determine whether or notthe detected regions are adjacent in the horizontal direction, therebydetecting connection of regions where arc shapes are formed,corresponding to the length-wise direction of the fine line image whichis signals of the actual world 1.

Also, the peak detecting unit 202 through continuousness detecting unit204 detect a region made up of pixels upon which the fine line image hasbeen projected wherein the same cross-sectional shape is arrayed in thehorizontal direction in the screen at constant intervals, and further,detect a region made up of pixels upon which the fine line image hasbeen projected which is a region having data continuity, by detectingconnection of detected regions corresponding to the length-wisedirection of the fine line of the actual world 1. That is to say, thepeak detecting unit 202 through continuousness detecting unit 204 detectregions wherein arc shapes are formed on a single horizontal row ofpixels in the input image, and determine whether or not the detectedregions are adjacent in the vertical direction, thereby detectingconnection of regions where arc shapes are formed, corresponding to thelength-wise direction of the fine line image, which is signals of theactual world 1.

First, description will be made regarding processing for detecting aregion of pixels upon which the fine line image has been projectedwherein the same arc shape is arrayed in the vertical direction in thescreen at constant intervals.

The peak detecting unit 202 detects a pixel having a pixel value greaterthan the surrounding pixels, i.e., a peak, and supplies peak informationindicating the position of the peak to the monotonous increase/decreasedetecting unit 203. In the event that pixels arrayed in a singlevertical direction row in the screen are the object, the peak detectingunit 202 compares the pixel value of the pixel position upwards in thescreen and the pixel value of the pixel position downwards in thescreen, and detects the pixel with the greater pixel value as the peak.The peak detecting unit 202 detects one or multiple peaks from a singleimage, e.g., from the image of a single frame.

A single screen contains frames or fields. This holds true in thefollowing description as well.

For example, the peak detecting unit 202 selects a pixel of interestfrom pixels of an image of one frame which have not yet been taken aspixels of interest, compares the pixel value of the pixel of interestwith the pixel value of the pixel above the pixel of interest, comparesthe pixel value of the pixel of interest with the pixel value of thepixel below the pixel of interest, detects a pixel of interest which hasa greater pixel value than the pixel value of the pixel above and agreater pixel value than the pixel value of the pixel below, and takesthe detected pixel of interest as a peak. The peak detecting unit 202supplies peak information indicating the detected peak to the monotonousincrease/decrease detecting unit 203.

There are cases wherein the peak detecting unit 202 does not detect apeak. For example, in the event that the pixel values of all of thepixels of an image are the same value, or in the event that the pixelvalues decrease in one or two directions, no peak is detected. In thiscase, no fine line image has been projected on the image data.

The monotonous increase/decrease detecting unit 203 detects a candidatefor a region made up of pixels upon which the fine line image has beenprojected wherein the pixels are vertically arrayed in a single row asto the peak detected by the peak detecting unit 202, based upon the peakinformation indicating the position of the peak supplied from the peakdetecting unit 202, and supplies the region information indicating thedetected region to the continuousness detecting unit 204 along with thepeak information.

More specifically, the monotonous increase/decrease detecting unit 203detects a region made up of pixels having pixel values monotonouslydecreasing with reference to the peak pixel value, as a candidate of aregion made up of pixels upon which the image of the fine line has beenprojected. Monotonous decrease means that the pixel values of pixelswhich are farther distance-wise from the peak are smaller than the pixelvalues of pixels which are closer to the peak.

Also, the monotonous increase/decrease detecting unit 203 detects aregion made up of pixels having pixel values monotonously increasingwith reference to the peak pixel value, as a candidate of a region madeup of pixels upon which the image of the fine line has been projected.Monotonous increase means that the pixel values of pixels which arefarther distance-wise from the peak are greater than the pixel values ofpixels which are closer to the peak.

In the following, the processing regarding regions of pixels havingpixel values monotonously increasing is the same as the processingregarding regions of pixels having pixel values monotonously decreasing,so description thereof will be omitted. Also, with the descriptionregarding processing for detecting a region of pixels upon which thefine line image has been projected wherein the same arc shape is arrayedhorizontally in the screen at constant intervals, the processingregarding regions of pixels having pixel values monotonously increasingis the same as the processing regarding regions of pixels having pixelvalues monotonously decreasing, so description thereof will be omitted.

For example, the monotonous increase/decrease detecting unit 203 obtainspixel values of each of the pixels in a vertical row as to a peak, thedifference as to the pixel value of the pixel above, and the differenceas to the pixel value of the pixel below. The monotonousincrease/decrease detecting unit 203 then detects a region wherein thepixel value monotonously decreases by detecting pixels wherein the signof the difference changes.

Further, the monotonous increase/decrease detecting unit 203 detects,from the region wherein pixel values monotonously decrease, a regionmade up of pixels having pixel values with the same sign as that of thepixel value of the peak, with the sign of the pixel value of the peak asa reference, as a candidate of a region made up of pixels upon which theimage of the fine line has been projected.

For example, the monotonous increase/decrease detecting unit 203compares the sign of the pixel value of each pixel with the sign of thepixel value of the pixel above and sign of the pixel value of the pixelbelow, and detects the pixel where the sign of the pixel value changes,thereby detecting a region of pixels having pixel values of the samesign as the peak within the region where pixel values monotonouslydecrease.

Thus, the monotonous increase/decrease detecting unit 203 detects aregion formed of pixels arrayed in a vertical direction wherein thepixel values monotonously decrease as to the peak and have pixels valuesof the same sign as the peak.

FIG. 36 is a diagram describing processing for peak detection andmonotonous increase/decrease region detection, for detecting the regionof pixels wherein the image of the fine line has been projected, fromthe pixel values as to a position in the spatial direction Y.

In FIG. 36 through FIG. 38, P represents a peak. In the description ofthe data continuity detecting unit 101 of which the configuration isshown in FIG. 30, P represents a peak.

The peak detecting unit 202 compares the pixel values of the pixels withthe pixel values of the pixels adjacent thereto in the spatial directionY, and detects the peak P by detecting a pixel having a pixel valuegreater than the pixel values of the two pixels adjacent in the spatialdirection Y.

The region made up of the peak P and the pixels on both sides of thepeak P in the spatial direction Y is a monotonous decrease regionwherein the pixel values of the pixels on both sides in the spatialdirection Y monotonously decrease as to the pixel value of the peak P.In FIG. 36, the arrow denoted A and the arrow denoted by B represent themonotonous decrease regions existing on either side of the peak P.

The monotonous increase/decrease detecting unit 203 obtains thedifference between the pixel values of each pixel and the pixel valuesof the pixels adjacent in the spatial direction Y, and detects pixelswhere the sign of the difference changes. The monotonousincrease/decrease detecting unit 203 takes the boundary between thedetected pixel where the sign of the difference changes and the pixelimmediately prior thereto (on the peak P side) as the boundary of thefine line region made up of pixels where the image of the fine line hasbeen projected.

In FIG. 36, the boundary of the fine line region which is the boundarybetween the pixel where the sign of the difference changes and the pixelimmediately prior thereto (on the peak P side) is denoted by C.

Further, the monotonous increase/decrease detecting unit 203 comparesthe sign of the pixel values of each pixel with the pixel values of thepixels adjacent thereto in the spatial direction Y, and detects pixelswhere the sign of the pixel value changes in the monotonous decreaseregion. The monotonous increase/decrease detecting unit 203 takes theboundary between the detected pixel where the sign of the pixel valuechanges and the pixel immediately prior thereto (on the peak P side) asthe boundary of the fine line region.

In FIG. 36, the boundary of the fine line region which is the boundarybetween the pixel where the sign of the pixel value changes and thepixel immediately prior thereto (on the peak P side) is denoted by D.

As shown in FIG. 36, the fine line region F made up of pixels where theimage of the fine line has been projected is the region between the fineline region boundary C and the fine line region boundary D.

The monotonous increase/decrease detecting unit 203 obtains a fine lineregion F which is longer than a predetermined threshold value, from fineline regions F made up of such monotonous increase/decrease regions,i.e., a fine line region F having a greater number of pixels than thethreshold value. For example, in the event that the threshold value is3, the monotonous increase/decrease detecting unit 203 detects a fineline region F including 4 or more pixels.

Further, the monotonous increase/decrease detecting unit 203 comparesthe pixel value of the peak P, the pixel value of the pixel to the rightside of the peak P, and the pixel value of the pixel to the left side ofthe peak P, from the fine line region F thus detected, each with thethreshold value, detects a fine line pixel region F having the peak Pwherein the pixel value of the peak P exceeds the threshold value, andwherein the pixel value of the pixel to the right side of the peak P isthe threshold value or lower, and wherein the pixel value of the pixelto the left side of the peak P is the threshold value or lower, andtakes the detected fine line region F as a candidate for the region madeup of pixels containing the component of the fine line image.

In other words, determination is made that a fine line region F havingthe peak P, wherein the pixel value of the peak P is the threshold valueor lower, or wherein the pixel value of the pixel to the right side ofthe peak P exceeds the threshold value, or wherein the pixel value ofthe pixel to the left side of the peak P exceeds the threshold value,does not contain the component of the fine line image, and is eliminatedfrom candidates for the region made up of pixels including the componentof the fine line image.

That is, as shown in FIG. 37, the monotonous increase/decrease detectingunit 203 compares the pixel value of the peak P with the thresholdvalue, and also compares the pixel value of the pixel adjacent to thepeak P in the spatial direction X (the direction indicated by the dottedline AA′) with the threshold value, thereby detecting the fine lineregion F to which the peak P belongs, wherein the pixel value of thepeak P exceeds the threshold value and wherein the pixel values of thepixel adjacent thereto in the spatial direction X are equal to or belowthe threshold value.

FIG. 38 is a diagram illustrating the pixel values of pixels arrayed inthe spatial direction X indicated by the dotted line AA′ in FIG. 37. Thefine line region F to which the peak P belongs, wherein the pixel valueof the peak P exceeds the threshold value Th_(s) and wherein the pixelvalues of the pixel adjacent thereto in the spatial direction X areequal to or below the threshold value Th_(s), contains the fine linecomponent.

Note that an arrangement may be made wherein the monotonousincrease/decrease detecting unit 203 compares the difference between thepixel value of the peak P and the pixel value of the background with thethreshold value, taking the pixel value of the background as areference, and also compares the difference between the pixel value ofthe pixels adjacent to the peak P in the spatial direction X and thepixel value of the background with the threshold value, therebydetecting the fine line region F to which the peak P belongs, whereinthe difference between the pixel value of the peak P and the pixel valueof the background exceeds the threshold value, and wherein thedifference between the pixel value of the pixel adjacent in the spatialdirection X and the pixel value of the background is equal to or belowthe threshold value.

The monotonous increase/decrease detecting unit 203 supplies to thecontinuousness detecting unit 204 monotonous increase/decrease regioninformation indicating a region made up of pixels of which the pixelvalue monotonously decrease with the peak P as a reference and the signof the pixel value is the same as that of the peak P, wherein the peak Pexceeds the threshold value and wherein the pixel value of the pixel tothe right side of the peak P is equal to or below the threshold valueand the pixel value of the pixel to the left side of the peak P is equalto or below the threshold value.

In the event of detecting a region of pixels arrayed in a single row inthe vertical direction of the screen where the image of the fine linehas been projected, pixels belonging to the region indicated by themonotonous increase/decrease region information are arrayed in thevertical direction and include pixels where the image of the fine linehas been projected. That is to say, the region indicated by themonotonous increase/decrease region information includes a region formedof pixels arrayed in a single row in the vertical direction of thescreen where the image of the fine line has been projected.

In this way, the peak detecting unit 202 and the monotonousincrease/decrease detecting unit 203 detects a continuity region made upof pixels where the image of the fine line has been projected, employingthe nature that, of the pixels where the image of the fine line has beenprojected, change in the pixel values in the spatial direction Yapproximates Gaussian distribution.

Of the region made up of pixels arrayed in the vertical direction,indicated by the monotonous increase/decrease region informationsupplied from the monotonous increase/decrease detecting unit 203, thecontinuousness detecting unit 204 detects regions including pixelsadjacent in the horizontal direction, i.e., regions having similar pixelvalue change and duplicated in the vertical direction, as continuousregions, and outputs the peak information and data continuityinformation indicating the detected continuous regions. The datacontinuity information includes monotonous increase/decrease regioninformation, information indicating the connection of regions, and soforth.

Arc shapes are aligned at constant intervals in an adjacent manner withthe pixels where the fine line has been projected, so the detectedcontinuous regions include the pixels where the fine line has beenprojected.

The detected continuous regions include the pixels where arc shapes arealigned at constant intervals in an adjacent manner to which the fineline has been projected, so the detected continuous regions are taken asa continuity region, and the continuousness detecting unit 204 outputsdata continuity information indicating the detected continuous regions.

That is to say, the continuousness detecting unit 204 uses thecontinuity wherein arc shapes are aligned at constant intervals in anadjacent manner in the data 3 obtained by imaging the fine line, whichhas been generated due to the continuity of the image of the fine linein the actual world 1, the nature of the continuity being continuing inthe length direction, so as to further narrow down the candidates ofregions detected with the peak detecting unit 202 and the monotonousincrease/decrease detecting unit 203.

FIG. 39 is a diagram describing the processing for detecting thecontinuousness of monotonous increase/decrease regions.

As shown in FIG. 39, in the event that a fine line region F formed ofpixels aligned in a single row in the vertical direction of the screenincludes pixels adjacent in the horizontal direction, the continuousnessdetecting unit 204 determines that there is continuousness between thetwo monotonous increase/decrease regions, and in the event that pixelsadjacent in the horizontal direction are not included, determines thatthere is no continuousness between the two fine line regions F. Forexample, a fine line region F⁻¹ made up of pixels aligned in a singlerow in the vertical direction of the screen is determined to becontinuous to a fine line region F₀ made up of pixels aligned in asingle row in the vertical direction of the screen in the event ofcontaining a pixel adjacent to a pixel of the fine line region F₀ in thehorizontal direction. The fine line region F₀ made up of pixels alignedin a single row in the vertical direction of the screen is determined tobe continuous to a fine line region F₁ made up of pixels aligned in asingle row in the vertical direction of the screen in the event ofcontaining a pixel adjacent to a pixel of the fine line region F₁ in thehorizontal direction.

In this way, regions made up of pixels aligned in a single row in thevertical direction of the screen where the image of the fine line hasbeen projected are detected by the peak detecting unit 202 through thecontinuousness detecting unit 204.

As described above, the peak detecting unit 202 through thecontinuousness detecting unit 204 detect regions made up of pixelsaligned in a single row in the vertical direction of the screen wherethe image of the fine line has been projected, and further detectregions made up of pixels aligned in a single row in the horizontaldirection of the screen where the image of the fine line has beenprojected.

Note that the order of processing is not particularly restricted, andmay be executed in parallel, as a matter of course.

That is to say, the peak detecting unit 202, with regard to of pixelsaligned in a single row in the horizontal direction of the screen,detects as a peak a pixel which has a pixel value greater in comparisonwith the pixel value of the pixel situated to the left side on thescreen and the pixel value of the pixel situated to the right side onthe screen, and supplies peak information indicating the position of thedetected peak to the monotonous increase/decrease detecting unit 203.The peak detecting unit 202 detects one or multiple peaks from oneimage, for example, one frame image.

For example, the peak detecting unit 202 selects a pixel of interestfrom pixels in the one frame image which has not yet been taken as apixel of interest, compares the pixel value of the pixel of interestwith the pixel value of the pixel to the left side of the pixel ofinterest, compares the pixel value of the pixel of interest with thepixel value of the pixel to the right side of the pixel of interest,detects a pixel of interest having a pixel value greater than the pixelvalue of the pixel to the left side of the pixel of interest and havinga pixel value greater than the pixel value of the pixel to the rightside of the pixel of interest, and takes the detected pixel of interestas a peak. The peak detecting unit 202 supplies peak informationindicating the detected peak to the monotonous increase/decreasedetecting unit 203.

There are cases wherein the peak detecting unit 202 does not detect apeak.

The monotonous increase/decrease detecting unit 203 detects candidatesfor a region made up of pixels aligned in a single row in the horizontaldirection as to the peak detected by the peak detecting unit 202 whereinthe fine line image has been projected, and supplies the monotonousincrease/decrease region information indicating the detected region tothe continuousness detecting unit 204 along with the peak information.

More specifically, the monotonous increase/decrease detecting unit 203detects regions made up of pixels having pixel values monotonouslydecreasing with the pixel value of the peak as a reference, ascandidates of regions made up of pixels where the fine line image hasbeen projected.

For example, the monotonous increase/decrease detecting unit 203obtains, with regard to each pixel in a single row in the horizontaldirection as to the peak, the pixel value of each pixel, the differenceas to the pixel value of the pixel to the left side, and the differenceas to the pixel value of the pixel to the right side. The monotonousincrease/decrease detecting unit 203 then detects the region where thepixel value monotonously decreases by detecting the pixel where the signof the difference changes.

Further, the monotonous increase/decrease detecting unit 203 detects,from a region wherein pixel values monotonously decrease, a region madeup of pixels having pixel values with the same sign as the pixel valueas the sign of the pixel value of the peak, with reference to the signof the pixel value of the peak, as a candidate for a region made up ofpixels where the fine line image has been projected.

For example, the monotonous increase/decrease detecting unit 203compares the sign of the pixel value of each pixel with the sign of thepixel value of the pixel to the left side or with the sign of the pixelvalue of the pixel to the right side, and detects the pixel where thesign of the pixel value changes, thereby detecting a region made up ofpixels having pixel values with the same sign as the peak, from theregion where the pixel values monotonously decrease.

Thus, the monotonous increase/decrease detecting unit 203 detects aregion made up of pixels aligned in the horizontal direction and havingpixel values with the same sign as the peak wherein the pixel valuesmonotonously decrease as to the peak.

From a fine line region made up of such a monotonous increase/decreaseregion, the monotonous increase/decrease detecting unit 203 obtains afine line region longer than a threshold value set beforehand, i.e., afine line region having a greater number of pixels than the thresholdvalue.

Further, from the fine line region thus detected, the monotonousincrease/decrease detecting unit 203 compares the pixel value of thepeak, the pixel value of the pixel above the peak, and the pixel valueof the pixel below the peak, each with the threshold value, detects afine line region to which belongs a peak wherein the pixel value of thepeak exceeds the threshold value, the pixel value of the pixel above thepeak is within the threshold, and the pixel value of the pixel below thepeak is within the threshold, and takes the detected fine line region asa candidate for a region made up of pixels containing the fine lineimage component.

Another way of saying this is that fine line regions to which belongs apeak wherein the pixel value of the peak is within the threshold value,or the pixel value of the pixel above the peak exceeds the threshold, orthe pixel value of the pixel below the peak exceeds the threshold, aredetermined to not contain the fine line image component, and areeliminated from candidates of the region made up of pixels containingthe fine line image component.

Note that the monotonous increase/decrease detecting unit 203 may bearranged to take the background pixel value as a reference, compare thedifference between the pixel value of the peak and the pixel value ofthe background with the threshold value, and also to compare thedifference between the pixel value of the background and the pixelvalues of the pixels adjacent to the peak in the vertical direction withthe threshold value, and take a detected fine line region wherein thedifference between the pixel value of the peak and the pixel value ofthe background exceeds the threshold value, and the difference betweenthe pixel value of the background and the pixel value of the pixelsadjacent in the vertical direction is within the threshold, as acandidate for a region made up of pixels containing the fine line imagecomponent.

The monotonous increase/decrease detecting unit 203 supplies to thecontinuousness detecting unit 204 monotonous increase/decrease regioninformation indicating a region made up of pixels having a pixel valuesign which is the same as the peak and monotonously decreasing pixelvalues as to the peak as a reference, wherein the peak exceeds thethreshold value, and the pixel value of the pixel to the right side ofthe peak is within the threshold, and the pixel value of the pixel tothe left side of the peak is within the threshold.

In the event of detecting a region made up of pixels aligned in a singlerow in the horizontal direction of the screen wherein the image of thefine line has been projected, pixels belonging to the region indicatedby the monotonous increase/decrease region information include pixelsaligned in the horizontal direction wherein the image of the fine linehas been projected. That is to say, the region indicated by themonotonous increase/decrease region information includes a region madeup of pixels aligned in a single row in the horizontal direction of thescreen wherein the image of the fine line has been projected.

Of the regions made up of pixels aligned in the horizontal directionindicated in the monotonous increase/decrease region informationsupplied from the monotonous increase/decrease detecting unit 203, thecontinuousness detecting unit 204 detects regions including pixelsadjacent in the vertical direction, i.e., regions having similar pixelvalue change and which are repeated in the horizontal direction, ascontinuous regions, and outputs data continuity information indicatingthe peak information and the detected continuous regions. The datacontinuity information includes information indicating the connection ofthe regions.

At the pixels where the fine line has been projected, arc shapes arearrayed at constant intervals in an adjacent manner, so the detectedcontinuous regions include pixels where the fine line has beenprojected.

The detected continuous regions include pixels where arc shapes arearrayed at constant intervals in an adjacent manner wherein the fineline has been projected, so the detected continuous regions are taken asa continuity region, and the continuousness detecting unit 204 outputsdata continuity information indicating the detected continuous regions.

That is to say, the continuousness detecting unit 204 uses thecontinuity which is that the arc shapes are arrayed at constantintervals in an adjacent manner in the data 3 obtained by imaging thefine line, generated from the continuity of the image of the fine linein the actual world 1 which is continuation in the length direction, soas to further narrow down the candidates of regions detected by the peakdetecting unit 202 and the monotonous increase/decrease detecting unit203.

Thus, the data continuity detecting unit 101 is capable of detectingcontinuity contained in the data 3 which is the input image. That is tosay, the data continuity detecting unit 101 can detect continuity ofdata included in the data 3 which has been generated by the actual world1 image which is a fine line having been projected on the data 3. Thedata continuity detecting unit 101 detects, from the data 3, regionsmade up of pixels where the actual world 1 image which is a fine linehas been projected.

FIG. 40 is a diagram illustrating an example of other processing fordetecting regions having continuity, where a fine line image has beenprojected, with the data continuity detecting unit 101.

As shown in FIG. 40, the data continuity detecting unit 101 calculatesthe absolute value of difference of pixel values for each pixel andadjacent pixels. The calculated absolute values of difference are placedcorresponding to the pixels. For example, in a situation such as shownin FIG. 40 wherein there are pixels aligned which have respective pixelvalues of P0, P1, and P2, the data continuity detecting unit 101calculates the difference d0=P0−P1 and the difference d1=P1−P2. Further,the data continuity detecting unit 101 calculates the absolute values ofthe difference d0 and the difference d1.

In the event that the non-continuity component contained in the pixelvalues P0, P1, and P2 are identical, only values corresponding to thecomponent of the fine line are set to the difference d0 and thedifference d1.

Accordingly, of the absolute values of the differences placedcorresponding to the pixels, in the event that adjacent differencevalues are identical, the data continuity detecting unit 101 determinesthat the pixel corresponding to the absolute values of the twodifferences (the pixel between the two absolute values of difference)contains the component of the fine line. However, the data continuitydetecting unit 101 does not need to detect the fine line in the eventthat the absolute value of difference is small. For example, in theevent that the absolute value of difference is equal to or greater thana threshold value, the data continuity detecting unit 101 determinesthat the pixel contains a fine line component.

The data continuity detecting unit 101 can also detect fine lines with asimple method such as this.

FIG. 41 is a flowchart for describing continuity detection processing.

In step S201, the non-continuity component extracting unit 201 extractsnon-continuity component, which is portions other than the portion wherethe fine line has been projected, from the input image. Thenon-continuity component extracting unit 201 supplies non-continuitycomponent information indicating the extracted non-continuity component,along with the input image, to the peak detecting unit 202 and themonotonous increase/decrease detecting unit 203. Details of theprocessing for extracting the non-continuity component will be describedlater.

In step S202, the peak detecting unit 202 eliminates the non-continuitycomponent from the input image, based on the non-continuity componentinformation supplied from the non-continuity component extracting unit201, so as to leave only pixels including the continuity component inthe input image. Further, in step S202, the peak detecting unit 202detects peaks.

That is to say, in the event of executing processing with the verticaldirection of the screen as a reference, of the pixels containing thecontinuity component, the peak detecting unit 202 compares the pixelvalue of each pixel with the pixel values of the pixels above and below,and detects pixels having a greater pixel value than the pixel value ofthe pixel above and the pixel value of the pixel below, therebydetecting a peak. Also, in step S202, in the event of executingprocessing with the horizontal direction of the screen as a reference,of the pixels containing the continuity component, the peak detectingunit 202 compares the pixel value of each pixel with the pixel values ofthe pixels to the right side and left side, and detects pixels having agreater pixel value than the pixel value of the pixel to the right sideand the pixel value of the pixel to the left side, thereby detecting apeak.

The peak detecting unit 202 supplies the peak information indicating thedetected peaks to the monotonous increase/decrease detecting unit 203.

In step S203, the monotonous increase/decrease detecting unit 203eliminates the non-continuity component from the input image, based onthe non-continuity component information supplied from thenon-continuity component extracting unit 201, so as to leave only pixelsincluding the continuity component in the input image. Further, in stepS203, the monotonous increase/decrease detecting unit 203 detects theregion made up of pixels having data continuity, by detecting monotonousincrease/decrease as to the peak, based on peak information indicatingthe position of the peak, supplied from the peak detecting unit 202.

In the event of executing processing with the vertical direction of thescreen as a reference, the monotonous increase/decrease detecting unit203 detects monotonous increase/decrease made up of one row of pixelsaligned vertically where a single fine line image has been projected,based on the pixel value of the peak and the pixel values of the one rowof pixels aligned vertically as to the peak, thereby detecting a regionmade up of pixels having data continuity. That is to say, in step S203,in the event of executing processing with the vertical direction of thescreen as a reference, the monotonous increase/decrease detecting unit203 obtains, with regard to a peak and a row of pixels alignedvertically as to the peak, the difference between the pixel value ofeach pixel and the pixel value of a pixel above or below, therebydetecting a pixel where the sign of the difference changes. Also, withregard to a peak and a row of pixels aligned vertically as to the peak,the monotonous increase/decrease detecting unit 203 compares the sign ofthe pixel value of each pixel with the sign of the pixel value of apixel above or below, thereby detecting a pixel where the sign of thepixel value changes. Further, the monotonous increase/decrease detectingunit 203 compares pixel value of the peak and the pixel values of thepixels to the right side and to the left side of the peak with athreshold value, and detects a region made up of pixels wherein thepixel value of the peak exceeds the threshold value, and wherein thepixel values of the pixels to the right side and to the left side of thepeak are within the threshold.

The monotonous increase/decrease detecting unit 203 takes a regiondetected in this way as a monotonous increase/decrease region, andsupplies monotonous increase/decrease region information indicating themonotonous increase/decrease region to the continuousness detecting unit204.

In the event of executing processing with the horizontal direction ofthe screen as a reference, the monotonous increase/decrease detectingunit 203 detects monotonous increase/decrease made up of one row ofpixels aligned horizontally where a single fine line image has beenprojected, based on the pixel value of the peak and the pixel values ofthe one row of pixels aligned horizontally as to the peak, therebydetecting a region made up of pixels having data continuity. That is tosay, in step S203, in the event of executing processing with thehorizontal direction of the screen as a reference, the monotonousincrease/decrease detecting unit 203 obtains, with regard to a peak anda row of pixels aligned horizontally as to the peak, the differencebetween the pixel value of each pixel and the pixel value of a pixel tothe right side or to the left side, thereby detecting a pixel where thesign of the difference changes. Also, with regard to a peak and a row ofpixels aligned horizontally as to the peak, the monotonousincrease/decrease detecting unit 203 compares the sign of the pixelvalue of each pixel with the sign of the pixel value of a pixel to theright side or to the left side, thereby detecting a pixel where the signof the pixel value changes. Further, the monotonous increase/decreasedetecting unit 203 compares pixel value of the peak and the pixel valuesof the pixels to the upper side and to the lower side of the peak with athreshold value, and detects a region made up of pixels wherein thepixel value of the peak exceeds the threshold value, and wherein thepixel values of the pixels to the upper side and to the lower side ofthe peak are within the threshold.

The monotonous increase/decrease detecting unit 203 takes a regiondetected in this way as a monotonous increase/decrease region, andsupplies monotonous increase/decrease region information indicating themonotonous increase/decrease region to the continuousness detecting unit204.

In step S204, the monotonous increase/decrease detecting unit 203determines whether or not processing of all pixels has ended. Forexample, the non-continuity component extracting unit 201 detects peaksfor all pixels of a single screen (for example, frame, field, or thelike) of the input image, and whether or not a monotonousincrease/decrease region has been detected is determined.

In the event that determination is made in step S204 that processing ofall pixels has not ended, i.e., that there are still pixels which havenot been subjected to the processing of peak detection and detection ofmonotonous increase/decrease region, the flow returns to step S202, apixel which has not yet been subjected to the processing of peakdetection and detection of monotonous increase/decrease region isselected as an object of the processing, and the processing of peakdetection and detection of monotonous increase/decrease region arerepeated.

In the event that determination is made in step S204 that processing ofall pixels has ended, i.e., that peaks and monotonous increase/decreaseregions have been detected with regard to all pixels, the flow proceedsto step S205, where the continuousness detecting unit 204 detects thecontinuousness of detected regions, based on the monotonousincrease/decrease region information. For example, in the event thatmonotonous increase/decrease regions made up of one row of pixelsaligned in the vertical direction of the screen, indicated by monotonousincrease/decrease region information, include pixels adjacent in thehorizontal direction, the continuousness detecting unit 204 determinesthat there is continuousness between the two monotonousincrease/decrease regions, and in the event of not including pixelsadjacent in the horizontal direction, determines that there is nocontinuousness between the two monotonous increase/decrease regions. Forexample, in the event that monotonous increase/decrease regions made upof one row of pixels aligned in the horizontal direction of the screen,indicated by monotonous increase/decrease region information, includepixels adjacent in the vertical direction, the continuousness detectingunit 204 determines that there is continuousness between the twomonotonous increase/decrease regions, and in the event of not includingpixels adjacent in the vertical direction, determines that there is nocontinuousness between the two monotonous increase/decrease regions.

The continuousness detecting unit 204 takes the detected continuousregions as continuity regions having data continuity, and outputs datacontinuity information indicating the peak position and continuityregion. The data continuity information contains information indicatingthe connection of regions. The data continuity information output fromthe continuousness detecting unit 204 indicates the fine line region,which is the continuity region, made up of pixels where the actual world1 fine line image has been projected.

In step S206, a continuity direction detecting unit 205 determineswhether or not processing of all pixels has ended. That is to say, thecontinuity direction detecting unit 205 determines whether or not regioncontinuation has been detected with regard to all pixels of a certainframe of the input image.

In the event that determination is made in step S206 that processing ofall pixels has not yet ended, i.e., that there are still pixels whichhave not yet been taken as the object of detection of regioncontinuation, the flow returns to step S205, a pixel which has not yetbeen subjected to the processing of detection of region continuity isselected as the pixel for processing, and the processing for detectionof region continuity is repeated.

In the event that determination is made in step S206 that processing ofall pixels has ended, i.e., that all pixels have been taken as theobject of detection of region continuity, the processing ends.

Thus, the continuity contained in the data 3 which is the input image isdetected. That is to say, continuity of data included in the data 3which has been generated by the actual world 1 image which is a fineline having been projected on the data 3 is detected, and a regionhaving data continuity, which is made up of pixels on which the actualworld 1 image which is a fine line has been projected, is detected fromthe data 3.

Now, the data continuity detecting unit 101 of which the configurationis shown in FIG. 30 can detect time-directional data continuity, basedon the region having data continuity detected from the frame of the data3.

For example, as shown in FIG. 42, the continuousness detecting unit 204detects time-directional data continuity by connecting the edges of theregion having detected data continuity in frame #n, the region havingdetected data continuity in frame #n−1, and the region having detecteddata continuity in frame #n+1.

The frame #n−1 is a frame preceding the frame #n time-wise, and theframe #n+1 is a frame following the frame #n time-wise. That is to say,the frame #n−1, the frame #n, and the frame #n+1, are displayed on theorder of the frame #n−1, the frame #n, and the frame #n+1.

More specifically, in FIG. 42, G denotes a movement vector obtained byconnecting the one edge of the region having detected data continuity inframe #n, the region having detected data continuity in frame #n−1, andthe region having detected data continuity in frame #n+1, and G′ denotesa movement vector obtained by connecting the other edges of the regionshaving detected data continuity. The movement vector G and the movementvector G′ are an example of data continuity in the time direction.

Further, the data continuity detecting unit 101 of which theconfiguration is shown in FIG. 30 can output information indicating thelength of the region having data continuity as data continuityinformation.

FIG. 43 is a block diagram illustrating the configuration of thenon-continuity component extracting unit 201 which performs planarapproximation of the non-continuity component which is the portion ofthe image data which does not have data continuity, and extracts thenon-continuity component.

The non-continuity component extracting unit 201 of which theconfiguration is shown in FIG. 43 extracts blocks, which are made up ofa predetermined number of pixels, from the input image, performs planarapproximation of the blocks, so that the error between the block and aplanar value is below a predetermined threshold value, therebyextracting the non-continuity component.

The input image is supplied to a block extracting unit 221, and is alsooutput without change.

The block extracting unit 221 extracts blocks, which are made up of apredetermined number of pixels, from the input image. For example, theblock extracting unit 221 extracts a block made up of 7×7 pixels, andsupplies this to a planar approximation unit 222. For example, the blockextracting unit 221 moves the pixel serving as the center of the blockto be extracted in raster scan order, thereby sequentially extractingblocks from the input image.

The planar approximation unit 222 approximates the pixel values of apixel contained in the block on a predetermined plane. For example, theplanar approximation unit 222 approximates the pixel value of a pixelcontained in the block on a plane expressed by Expression (32).z=ax+by+c  (32)

In Expression (32), x represents the position of the pixel in onedirection on the screen (the spatial direction X), and y represents theposition of the pixel in the other direction on the screen (the spatialdirection Y). z represents the application value represented by theplane. a represents the gradient of the spatial direction X of theplane, and b represents the gradient of the spatial direction Y of theplane. In Expression (32), c represents the offset of the plane(intercept).

For example, the planar approximation unit 222 obtains the gradient a,gradient b, and offset c, by regression processing, therebyapproximating the pixel values of the pixels contained in the block on aplane expressed by Expression (32). The planar approximation unit 222obtains the gradient a, gradient b, and offset c, by regressionprocessing including rejection, thereby approximating the pixel valuesof the pixels contained in the block on a plane expressed by Expression(32).

For example, the planar approximation unit 222 obtains the planeexpressed by Expression (32) wherein the error is least as to the pixelvalues of the pixels of the block using the least-square method, therebyapproximating the pixel values of the pixels contained in the block onthe plane.

Note that while the planar approximation unit 222 has been describedapproximating the block on the plane expressed by Expression (32), thisis not restricted to the plane expressed by Expression (32), rather, theblock may be approximated on a plane represented with a function with ahigher degree of freedom, for example, an n-order (wherein n is anarbitrary integer) polynomial.

A repetition determining unit 223 calculates the error between theapproximation value represented by the plane upon which the pixel valuesof the block have been approximated, and the corresponding pixel valuesof the pixels of the block. Expression (33) is an expression which showsthe error ei which is the difference between the approximation valuerepresented by the plane upon which the pixel values of the block havebeen approximated, and the corresponding pixel values zi of the pixelsof the block.

$\begin{matrix}{e_{i} = {{z_{i} - \hat{z}} = {z_{i} - \left( {{\hat{a}\; x_{i\;}} + {\hat{b}\; y_{i}} + \hat{c}} \right)}}} & (33)\end{matrix}$

In Expression (33), z-hat (A symbol with ^ over z will be described asz-hat. The same description will be used in the present specificationhereafter.) represents an approximation value expressed by the plane onwhich the pixel values of the block are approximated, a-hat representsthe gradient of the spatial direction X of the plane on which the pixelvalues of the block are approximated, b-hat represents the gradient ofthe spatial direction Y of the plane on which the pixel values of theblock are approximated, and c-hat represents the offset (intercept) ofthe plane on which the pixel values of the block are approximated.

The repetition determining unit 223 rejects the pixel regarding whichthe error ei between the approximation value and the corresponding pixelvalues of pixels of the block is greatest, shown in Expression (33).Thus, pixels where the fine line has been projected, i.e., pixels havingcontinuity, are rejected. The repetition determining unit 223 suppliesrejection information indicating the rejected pixels to the planarapproximation unit 222.

Further, the repetition determining unit 223 calculates a standarderror, and in the event that the standard error is equal to or greaterthan threshold value which has been set beforehand for determiningending of approximation, and half or more of the pixels of the pixels ofa block have not been rejected, the repetition determining unit 223causes the planar approximation unit 222 to repeat the processing ofplanar approximation on the pixels contained in the block, from whichthe rejected pixels have been eliminated.

Pixels having continuity are rejected, so approximating the pixels fromwhich the rejected pixels have been eliminated on a plane means that theplane approximates the non-continuity component.

At the point that the standard error below the threshold value fordetermining ending of approximation, or half or more of the pixels ofthe pixels of a block have been rejected, the repetition determiningunit 223 ends planar approximation.

With a block made up of 5×5 pixels, the standard error e_(s) can becalculated with, for example, Expression (34).

$\begin{matrix}\begin{matrix}{e_{s} = {\sum{\left( {z_{i} - \hat{z}} \right)/\left( {n - 3} \right)}}} \\{= {\sum\left\{ {\left( {z_{i} - \left( {{\hat{a}\; x_{i\;}} + {\hat{b}\; y_{i}} + \hat{c}} \right)} \right\}/\left( {n - 3} \right)} \right.}}\end{matrix} & (34)\end{matrix}$

Here, n is the number of pixels.

Note that the repetition determining unit 223 is not restricted tostandard error, and may be arranged to calculate the sum of the squareof errors for all of the pixels contained in the block, and perform thefollowing processing.

Now, at the time of planar approximation of blocks shifted one pixel inthe raster scan direction, a pixel having continuity, indicated by theblack circle in the diagram, i.e., a pixel containing the fine linecomponent, will be rejected multiple times, as shown in FIG. 44.

Upon completing planar approximation, the repetition determining unit223 outputs information expressing the plane for approximating the pixelvalues of the block (the gradient and intercept of the plane ofExpression 32)) as non-continuity information.

Note that an arrangement may be made wherein the repetition determiningunit 223 compares the number of times of rejection per pixel with apreset threshold value, and takes a pixel which has been rejected anumber of times equal to or greater than the threshold value as a pixelcontaining the continuity component, and output the informationindicating the pixel including the continuity component as continuitycomponent information. In this case, the peak detecting unit 202 throughthe continuity direction detecting unit 205 execute their respectiveprocessing on pixels containing continuity component, indicated by thecontinuity component information.

The number of times of rejection, the gradient of the spatial directionX of the plane for approximating the pixel values of the pixel of theblock, the gradient of the spatial direction Y of the plane forapproximating the pixel values of the pixel of the block, approximationvalues expressed by the plane approximating the pixel values of thepixels of the block, and the error ei, can be used as features of theinput image.

FIG. 45 is a flowchart for describing the processing of extracting thenon-continuity component with the non-continuity component extractingunit 201 of which the configuration is shown in FIG. 43, correspondingto step S201.

In step S221, the block extracting unit 221 extracts a block made up ofa predetermined number of pixels from the input image, and supplies theextracted block to the planar approximation unit 222. For example, theblock extracting unit 221 selects one pixel of the pixels of the inputpixel which have not been selected yet, and extracts a block made up of7×7 pixels centered on the selected pixel. For example, the blockextracting unit 221 can select pixels in raster scan order.

In step S222, the planar approximation unit 222 approximates theextracted block on a plane. The planar approximation unit 222approximates the pixel values of the pixels of the extracted block on aplane by regression processing, for example. For example, the planarapproximation unit 222 approximates the pixel values of the pixels ofthe extracted block excluding the rejected pixels on a plane, byregression processing. In step S223, the repetition determining unit 223executes repetition determination. For example, repetition determinationis performed by calculating the standard error from the pixel values ofthe pixels of the block and the planar approximation values, andcounting the number of rejected pixels.

In step S224, the repetition determining unit 223 determines whether ornot the standard error is equal to or above a threshold value, and inthe event that determination is made that the standard error is equal toor above the threshold value, the flow proceeds to step S225.

Note that an arrangement may be made wherein the repetition determiningunit 223 determines in step S224 whether or not half or more of thepixels of the block have been rejected, and whether or not the standarderror is equal to or above the threshold value, and in the event thatdetermination is made that half or more of the pixels of the block havenot been rejected, and the standard error is equal to or above thethreshold value, the flow proceeds to step S225.

In step S225, the repetition determining unit 223 calculates the errorbetween the pixel value of each pixel of the block and the approximatedplanar approximation value, rejects the pixel with the greatest error,and notifies the planar approximation unit 222. The procedure returns tostep S222, and the planar approximation processing and repetitiondetermination processing is repeated with regard to the pixels of theblock excluding the rejected pixel.

In step S225, in the event that a block which is shifted one pixel inthe raster scan direction is extracted in the processing in step S221,the pixel including the fine line component (indicated by the blackcircle in the drawing) is rejected multiple times, as shown in FIG. 44.

In the event that determination is made in step S224 that the standarderror is not equal to or greater than the threshold value, the block hasbeen approximated on the plane, so the flow proceeds to step S226.

Note that an arrangement may be made wherein the repetition determiningunit 223 determines in step S224 whether or not half or more of thepixels of the block have been rejected, and whether or not the standarderror is equal to or above the threshold value, and in the event thatdetermination is made that half or more of the pixels of the block havebeen rejected, or the standard error is not equal to or above thethreshold value, the flow proceeds to step S225.

In step S226, the repetition determining unit 223 outputs the gradientand intercept of the plane for approximating the pixel values of thepixels of the block as non-continuity component information.

In step S227, the block extracting unit 221 determines whether or notprocessing of all pixels of one screen of the input image has ended, andin the event that determination is made that there are still pixelswhich have not yet been taken as the object of processing, the flowreturns to step S221, a block is extracted from pixels not yet beensubjected to the processing, and the above processing is repeated.

In the event that determination is made in step S227 that processing hasended for all pixels of one screen of the input image, the processingends.

Thus, the non-continuity component extracting unit 201 of which theconfiguration is shown in FIG. 43 can extract the non-continuitycomponent from the input image. The non-continuity component extractingunit 201 extracts the non-continuity component from the input image, sothe peak detecting unit 202 and monotonous increase/decrease detectingunit 203 can obtain the difference between the input image and thenon-continuity component extracted by the non-continuity componentextracting unit 201, so as to execute the processing regarding thedifference containing the continuity component.

Note that the standard error in the event that rejection is performed,the standard error in the event that rejection is not performed, thenumber of times of rejection of a pixel, the gradient of the spatialdirection X of the plane (a-hat in Expression (32)), the gradient of thespatial direction Y of the plane (b-hat in Expression (32)), the levelof planar transposing (c-hat in Expression (32)), and the differencebetween the pixel values of the input image and the approximation valuesrepresented by the plane, calculated in planar approximation processing,can be used as features.

FIG. 46 is a flowchart for describing processing for extracting thecontinuity component with the non-continuity component extracting unit201 of which the configuration is shown in FIG. 43, instead of theprocessing for extracting the non-continuity component corresponding tostep S201. The processing of step S241 through step S245 is the same asthe processing of step S221 through step S225, so description thereofwill be omitted.

In step S246, the repetition determining unit 223 outputs the differencebetween the approximation value represented by the plane and the pixelvalues of the input image, as the continuity component of the inputimage. That is to say, the repetition determining unit 223 outputs thedifference between the planar approximation values and the true pixelvalues.

Note that the repetition determining unit 223 may be arranged to outputthe difference between the approximation value represented by the planeand the pixel values of the input image, regarding pixel values ofpixels of which the difference is equal to or greater than apredetermined threshold value, as the continuity component of the inputimage.

The processing of step S247 is the same as the processing of step S227,and accordingly description thereof will be omitted.

The plane approximates the non-continuity component, so thenon-continuity component extracting unit 201 can remove thenon-continuity component from the input image by subtracting theapproximation value represented by the plane for approximating pixelvalues, from the pixel values of each pixel in the input image. In thiscase, the peak detecting unit 202 through the continuousness detectingunit 204 can be made to process only the continuity component of theinput image, i.e., the values where the fine line image has beenprojected, so the processing with the peak detecting unit 202 throughthe continuousness detecting unit 204 becomes easier.

FIG. 47 is a flowchart for describing other processing for extractingthe continuity component with the non-continuity component extractingunit 201 of which the configuration is shown in FIG. 43, instead of theprocessing for extracting the non-continuity component corresponding tostep S201. The processing of step S261 through step S265 is the same asthe processing of step S221 through step S225, so description thereofwill be omitted.

In step S266, the repetition determining unit 223 stores the number oftimes of rejection for each pixel, the flow returns to step S262, andthe processing is repeated.

In step S264, in the event that determination is made that the standarderror is not equal to or greater than the threshold value, the block hasbeen approximated on the plane, so the flow proceeds to step S267, therepetition determining unit 223 determines whether or not processing ofall pixels of one screen of the input image has ended, and in the eventthat determination is made that there are still pixels which have notyet been taken as the object of processing, the flow returns to stepS261, with regard to a pixel which has not yet been subjected to theprocessing, a block is extracted, and the above processing is repeated.

In the event that determination is made in step S267 that processing hasended for all pixels of one screen of the input image, the flow proceedsto step S268, the repetition determining unit 223 selects a pixel whichhas not yet been selected, and determines whether or not the number oftimes of rejection of the selected pixel is equal to or greater than athreshold value. For example, the repetition determining unit 223determines in step S268 whether or not the number of times of rejectionof the selected pixel is equal to or greater than a threshold valuestored beforehand.

In the event that determination is made in step S268 that the number oftimes of rejection of the selected pixel is equal to or greater than thethreshold value, the selected pixel contains the continuity component,so the flow proceeds to step S269, where the repetition determining unit223 outputs the pixel value of the selected pixel (the pixel value inthe input image) as the continuity component of the input image, and theflow proceeds to step S270.

In the event that determination is made in step S268 that the number oftimes of rejection of the selected pixel is not equal to or greater thanthe threshold value, the selected pixel does not contain the continuitycomponent, so the processing in step S269 is skipped, and the procedureproceeds to step S270. That is to say, the pixel value of a pixelregarding which determination has been made that the number of times ofrejection is not equal to or greater than the threshold value is notoutput.

Note that an arrangement may be made wherein the repetition determiningunit 223 outputs a pixel value set to 0 for pixels regarding whichdetermination has been made that the number of times of rejection is notequal to or greater than the threshold value.

In step S270, the repetition determining unit 223 determines whether ornot processing of all pixels of one screen of the input image has endedto determine whether or not the number of times of rejection is equal toor greater than the threshold value, and in the event that determinationis made that processing has not ended for all pixels, this means thatthere are still pixels which have not yet been taken as the object ofprocessing, so the flow returns to step S268, a pixel which has not yetbeen subjected to the processing is selected, and the above processingis repeated.

In the event that determination is made in step S270 that processing hasended for all pixels of one screen of the input image, the processingends.

Thus, of the pixels of the input image, the non-continuity componentextracting unit 201 can output the pixel values of pixels containing thecontinuity component, as continuity component information. That is tosay, of the pixels of the input image, the non-continuity componentextracting unit 201 can output the pixel values of pixels containing thecomponent of the fine line image.

FIG. 48 is a flowchart for describing yet other processing forextracting the continuity component with the non-continuity componentextracting unit 201 of which the configuration is shown in FIG. 43,instead of the processing for extracting the non-continuity componentcorresponding to step S201. The processing of step S281 through stepS288 is the same as the processing of step S261 through step S268, sodescription thereof will be omitted.

In step S289, the repetition determining unit 223 outputs the differencebetween the approximation value represented by the plane, and the pixelvalue of a selected pixel, as the continuity component of the inputimage. That is to say, the repetition determining unit 223 outputs animage wherein the non-continuity component has been removed from theinput image, as the continuity information.

The processing of step S290 is the same as the processing of step S270,and accordingly description thereof will be omitted.

Thus, the non-continuity component extracting unit 201 can output animage wherein the non-continuity component has been removed from theinput image as the continuity information.

As described above, in a case wherein real world light signals areprojected, a non-continuous portion of pixel values of multiple pixelsof first image data wherein a part of the continuity of the real worldlight signals has been lost is detected, data continuity is detectedfrom the detected non-continuous portions, a model (function) isgenerated for approximating the light signals by estimating thecontinuity of the real world light signals based on the detected datacontinuity, and second image data is generated based on the generatedfunction, processing results which are more accurate and have higherprecision as to the event in the real world can be obtained.

FIG. 49 is a block diagram illustrating another configuration of thedata continuity detecting unit 101.

With the data continuity detecting unit 101 of which the configurationis shown in FIG. 49, change in the pixel value of the pixel of interestwhich is a pixel of interest in the spatial direction of the inputimage, i.e. activity in the spatial direction of the input image, isdetected, multiple sets of pixels made up of a predetermined number ofpixels in one row in the vertical direction or one row in the horizontaldirection are extracted for each angle based on the pixel of interestand a reference axis according to the detected activity, the correlationof the extracted pixel sets is detected, and the angle of datacontinuity based on the reference axis in the input image is detectedbased on the correlation.

The angle of data continuity means an angle assumed by the referenceaxis, and the direction of a predetermined dimension where continuityrepeatedly appear in the data 3. Continuity repeatedly appearing means acase wherein, for example, the change in value as to the change inposition in the data 3, i.e., the cross-sectional shape, is the same,and so forth.

The reference axis may be, for example, an axis indicating the spatialdirection X (the horizontal direction of the screen), an axis indicatingthe spatial direction Y (the vertical direction of the screen), and soforth.

The input image is supplied to an activity detecting unit 401 and dataselecting unit 402.

The activity detecting unit 401 detects change in the pixel values as tothe spatial direction of the input image, i.e., activity in the spatialdirection, and supplies the activity information which indicates thedetected results to the data selecting unit 402 and a continuitydirection derivation unit 404.

For example, the activity detecting unit 401 detects the change of apixel value as to the horizontal direction of the screen, and the changeof a pixel value as to the vertical direction of the screen, andcompares the detected change of the pixel value in the horizontaldirection and the change of the pixel value in the vertical direction,thereby detecting whether the change of the pixel value in thehorizontal direction is greater as compared with the change of the pixelvalue in the vertical direction, or whether the change of the pixelvalue in the vertical direction is greater as compared with the changeof the pixel value in the horizontal direction.

The activity detecting unit 401 supplies to the data selecting unit 402and the continuity direction derivation unit 404 activity information,which is the detection results, indicating that the change of the pixelvalue in the horizontal direction is greater as compared with the changeof the pixel value in the vertical direction, or indicating that thechange of the pixel value in the vertical direction is greater ascompared with the change of the pixel value in the horizontal direction.

In the event that the change of the pixel value in the horizontaldirection is greater as compared with the change of the pixel value inthe vertical direction, arc shapes (half-disc shapes) or pawl shapes areformed on one row of pixels in the vertical direction, as indicated byFIG. 50 for example, and the arc shapes or pawl shapes are formedrepetitively more in the vertical direction. That is to say, in theevent that the change of the pixel value in the horizontal direction isgreater as compared with the change of the pixel value in the verticaldirection, with the reference axis as the axis representing the spatialdirection X, the angle of the data continuity based on the referenceaxis in the input image is a value of any from 45 degrees to 90 degrees.

In the event that the change of the pixel value in the verticaldirection is greater as compared with the change of the pixel value inthe horizontal direction, arc shapes or pawl shapes are formed on onerow of pixels in the vertical direction, for example, and the arc shapesor pawl shapes are formed repetitively more in the horizontal direction.That is to say, in the event that the change of the pixel value in thevertical direction is greater as compared with the change of the pixelvalue in the horizontal direction, with the reference axis as the axisrepresenting the spatial direction X, the angle of the data continuitybased on the reference axis in the input image is a value of any from 0degrees to 45 degrees.

For example, the activity detecting unit 401 extracts from the inputimage a block made up of the 9 pixels, 3×3 centered on the pixel ofinterest, as shown in FIG. 51. The activity detecting unit 401calculates the sum of differences of the pixels values regarding thepixels vertically adjacent, and the sum of differences of the pixelsvalues regarding the pixels horizontally adjacent. The sum ofdifferences h_(diff) of the pixels values regarding the pixelshorizontally adjacent can be obtained with Expression (35).h _(diff)=Σ(P _(i+1,j) −P _(i,j))  (35)

In the same way, the sum of differences v_(diff) of the pixels valuesregarding the pixels vertically adjacent can be obtained with Expression(36).v _(diff)=Σ(P _(i,j+1) −P _(i,j))  (36)

In Expression (35) and Expression (36), P represents the pixel value, irepresents the position of the pixel in the horizontal direction, and jrepresents the position of the pixel in the vertical direction.

An arrangement may be made wherein the activity detecting unit 401compares the calculated sum of differences h_(diff) of the pixels valuesregarding the pixels horizontally adjacent with the sum of differencesv_(diff) of the pixels values regarding the pixels vertically adjacent,so as to determine the range of the angle of the data continuity basedon the reference axis in the input image. That is to say, in this case,the activity detecting unit 401 determines whether a shape indicated bychange in the pixel value as to the position in the spatial direction isformed repeatedly in the horizontal direction, or formed repeatedly inthe vertical direction.

For example, change in pixel values in the horizontal direction withregard to an arc formed on pixels in one horizontal row is greater thanthe change of pixel values in the vertical direction, change in pixelvalues in the vertical direction with regard to an arc formed on pixelsin one horizontal row is greater than the change of pixel values in thehorizontal direction, and it can be said that the direction of datacontinuity, i.e., the change in the direction of the predetermineddimension of a constant feature which the input image that is the data 3has is smaller in comparison with the change in the orthogonal directiontoo the data continuity. In other words, the difference of the directionorthogonal to the direction of data continuity (hereafter also referredto as non-continuity direction) is greater as compared to the differencein the direction of data continuity.

For example, as shown in FIG. 52, the activity detecting unit 401compares the calculated sum of differences h_(diff) of the pixels valuesregarding the pixels horizontally adjacent with the sum of differencesv_(diff) of the pixels values regarding the pixels vertically adjacent,and in the event that the sum of differences h_(diff) of the pixelsvalues regarding the pixels horizontally adjacent is greater, determinesthat the angle of the data continuity based on the reference axis is avalue of any from 45 degrees to 135 degrees, and in the event that thesum of differences v_(diff) of the pixels values regarding the pixelsvertically adjacent is greater, determines that the angle of the datacontinuity based on the reference axis is a value of any from 0 degreesto 45 degrees, or a value of any from 135 degrees to 180 degrees.

For example, the activity detecting unit 401 supplies activityinformation indicating the determination results to the data selectingunit 402 and the continuity direction derivation unit 404.

Note that the activity detecting unit 401 can detect activity byextracting blocks of arbitrary sizes, such as a block made up of 25pixels of 5×5, a block made up of 49 pixels of 7×7, and so forth.

The data selecting unit 402 sequentially selects pixels of interest fromthe pixels of the input image, and extracts multiple sets of pixels madeup of a predetermined number of pixels in one row in the verticaldirection or one row in the horizontal direction for each angle based onthe pixel of interest and the reference axis, based on the activityinformation supplied from the activity detecting unit 401.

For example, in the event that the activity information indicates thatthe change in pixel values in the horizontal direction is greater incomparison with the change in pixel values in the vertical direction,this means that the data continuity angle is a value of any from 45degrees to 135 degrees, so the data selecting unit 402 extracts multiplesets of pixels made up of a predetermined number of pixels in one row inthe vertical direction, for each predetermined angle in the range of 45degrees to 135 degrees, based on the pixel of interest and the referenceaxis.

In the event that the activity information indicates that the change inpixel values in the vertical direction is greater in comparison with thechange in pixel values in the horizontal direction, this means that thedata continuity angle is a value of any from 0 degrees to 45 degrees orfrom 135 degrees to 180 degrees, so the data selecting unit 402 extractsmultiple sets of pixels made up of a predetermined number of pixels inone row in the horizontal direction, for each predetermined angle in therange of 0 degrees to 45 degrees or 135 degrees to 180 degrees, based onthe pixel of interest and the reference axis.

Also, for example, in the event that the activity information indicatesthat the angle of data continuity is a value of any from 45 degrees to135 degrees, the data selecting unit 402 extracts multiple sets ofpixels made up of a predetermined number of pixels in one row in thevertical direction, for each predetermined angle in the range of 45degrees to 135 degrees, based on the pixel of interest and the referenceaxis.

In the event that the activity information indicates that the angle ofdata continuity is a value of any from 0 degrees to 45 degrees or from135 degrees to 180 degrees, the data selecting unit 402 extractsmultiple sets of pixels made up of a predetermined number of pixels inone row in the horizontal direction, for each predetermined angle in therange of 0 degrees to 45 degrees or 135 degrees to 180 degrees, based onthe pixel of interest and the reference axis.

The data selecting unit 402 supplies the multiple sets made up of theextracted pixels to an error estimating unit 403.

The error estimating unit 403 detects correlation of pixel sets for eachangle with regard to the multiple sets of extracted pixels.

For example, with regard to the multiple sets of pixels made up of apredetermined number of pixels in one row in the vertical directioncorresponding to one angle, the error estimating unit 403 detects thecorrelation of the pixels values of the pixels at correspondingpositions of the pixel sets. With regard to the multiple sets of pixelsmade up of a predetermined number of pixels in one row in the horizontaldirection corresponding to one angle, the error estimating unit 403detects the correlation of the pixels values of the pixels atcorresponding positions of the sets.

The error estimating unit 403 supplies correlation informationindicating the detected correlation to the continuity directionderivation unit 404. The error estimating unit 403 calculates the sum ofthe pixel values of pixels of a set including the pixel of interestsupplied from the data selecting unit 402 as values indicatingcorrelation, and the absolute value of difference of the pixel values ofthe pixels at corresponding positions in other sets, and supplies thesum of absolute value of difference to the continuity directionderivation unit 404 as correlation information.

Based on the correlation information supplied from the error estimatingunit 403, the continuity direction derivation unit 404 detects the datacontinuity angle based on the reference axis in the input image,corresponding to the lost continuity of the light signals of the actualworld 1, and outputs data continuity information indicating an angle.For example, based on the correlation information supplied from theerror estimating unit 403, the continuity direction derivation unit 404detects an angle corresponding to the pixel set with the greatestcorrelation as the data continuity angle, and outputs data continuityinformation indicating the angle corresponding to the pixel set with thegreatest correlation that has been detected.

The following description will be made regarding detection of datacontinuity angle in the range of 0 degrees through 90 degrees (theso-called first quadrant).

FIG. 53 is a block diagram illustrating a more detailed configuration ofthe data continuity detecting unit 101 shown in FIG. 49.

The data selecting unit 402 includes pixel selecting unit 411-1 throughpixel selecting unit 411-L. The error estimating unit 403 includesestimated error calculating unit 412-1 through estimated errorcalculating unit 412-L. The continuity direction derivation unit 404includes a smallest error angle selecting unit 413.

First, description will be made regarding the processing of the pixelselecting unit 411-1 through pixel selecting unit 411-L in the eventthat the data continuity angle indicated by the activity information isa value of any from 45 degrees to 135 degrees.

The pixel selecting unit 411-1 through pixel selecting unit 411-L setstraight lines of mutually differing predetermined angles which passthrough the pixel of interest, with the axis indicating the spatialdirection X as the reference axis. The pixel selecting unit 411-1through pixel selecting unit 411-L select, of the pixels belonging to avertical row of pixels to which the pixel of interest belongs, apredetermined number of pixels above the pixel of interest, andpredetermined number of pixels below the pixel of interest, and thepixel of interest, as a set.

For example, as shown in FIG. 54, the pixel selecting unit 411-1 throughpixel selecting unit 411-L select 9 pixels centered on the pixel ofinterest, as a set of pixels, from the pixels belonging to a verticalrow of pixels to which the pixel of interest belongs.

In FIG. 54, one grid-shaped square (one grid) represents one pixel. InFIG. 54, the circle shown at the center represents the pixel ofinterest.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a vertical row of pixels to the left ofthe vertical row of pixels to which the pixel of interest belongs, apixel at the position closest to the straight line set for each. In FIG.54, the circle to the lower left of the pixel of interest represents anexample of a selected pixel. The pixel selecting unit 411-1 throughpixel selecting unit 411-L then select, from the pixels belonging to thevertical row of pixels to the left of the vertical row of pixels towhich the pixel of interest belongs, a predetermined number of pixelsabove the selected pixel, a predetermined number of pixels below theselected pixel, and the selected pixel, as a set of pixels.

For example, as shown in FIG. 54, the pixel selecting unit 411-1 throughpixel selecting unit 411-L select 9 pixels centered on the pixel at theposition closest to the straight line, from the pixels belonging to thevertical row of pixels to the left of the vertical row of pixels towhich the pixel of interest belongs, as a set of pixels.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a vertical row of pixels second leftfrom the vertical row of pixels to which the pixel of interest belongs,a pixel at the position closest to the straight line set for each. InFIG. 54, the circle to the far left represents an example of theselected pixel. The pixel selecting unit 411-1 through pixel selectingunit 411-L then select, as a set of pixels, from the pixels belonging tothe vertical row of pixels second left from the vertical row of pixelsto which the pixel of interest belongs, a predetermined number of pixelsabove the selected pixel, a predetermined number of pixels below theselected pixel, and the selected pixel.

For example, as shown in FIG. 54, the pixel selecting unit 411-1 throughpixel selecting unit 411-L select 9 pixels centered on the pixel at theposition closest to the straight line, from the pixels belonging to thevertical row of pixels second left from the vertical row of pixels towhich the pixel of interest belongs, as a set of pixels.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a vertical row of pixels to the rightof the vertical row of pixels to which the pixel of interest belongs, apixel at the position closest to the straight line set for each. In FIG.54, the circle to the upper right of the pixel of interest represents anexample of a selected pixel. The pixel selecting unit 411-1 throughpixel selecting unit 411-L then select, from the pixels belonging to thevertical row of pixels to the right of the vertical row of pixels towhich the pixel of interest belongs, a predetermined number of pixelsabove the selected pixel, a predetermined number of pixels below theselected pixel, and the selected pixel, as a set of pixels.

For example, as shown in FIG. 54, the pixel selecting unit 411-1 throughpixel selecting unit 411-L select 9 pixels centered on the pixel at theposition closest to the straight line, from the pixels belonging to thevertical row of pixels to the right of the vertical row of pixels towhich the pixel of interest belongs, as a set of pixels.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a vertical row of pixels second rightfrom the vertical row of pixels to which the pixel of interest belongs,a pixel at the position closest to the straight line set for each. InFIG. 54, the circle to the far right represents an example of theselected pixel. The pixel selecting unit 411-1 through pixel selectingunit 411-L then select, from the pixels belonging to the vertical row ofpixels second right from the vertical row of pixels to which the pixelof interest belongs, a predetermined number of pixels above the selectedpixel, a predetermined number of pixels below the selected pixel, andthe selected pixel, as a set of pixels.

For example, as shown in FIG. 54, the pixel selecting unit 411-1 throughpixel selecting unit 411-L select 9 pixels centered on the pixel at theposition closest to the straight line, from the pixels belonging to thevertical row of pixels second right from the vertical row of pixels towhich the pixel of interest belongs, as a set of pixels.

Thus, the pixel selecting unit 411-1 through pixel selecting unit 411-Leach select five sets of pixels.

The pixel selecting unit 411-1 through pixel selecting unit 411-L selectpixel sets for (lines set to) mutually different angles. For example,the pixel selecting unit 411-1 selects sets of pixels regarding 45degrees, the pixel selecting unit 411-2 selects sets of pixels regarding47.5 degrees, and the pixel selecting unit 411-3 selects sets of pixelsregarding 50 degrees. The pixel selecting unit 411-1 through pixelselecting unit 411-L select sets of pixels regarding angles every 2.5degrees, from 52.5 degrees through 135 degrees.

Note that the number of pixel sets may be an optional number, such as 3or 7, for example, and does not restrict the present invention. Also,the number of pixels selected as one set may be an optional number, suchas 5 or 13, for example, and does not restrict the present invention.

Note that the pixel selecting unit 411-1 through pixel selecting unit411-L may be arranged to select pixel sets from pixels within apredetermined range in the vertical direction. For example, the pixelselecting unit 411-1 through pixel selecting unit 411-L can select pixelsets from 121 pixels in the vertical direction (60 pixels upward fromthe pixel of interest, and 60 pixels downward). In this case, the datacontinuity detecting unit 101 can detect the angle of data continuity upto 88.09 degrees as to the axis representing the spatial direction X.

The pixel selecting unit 411-1 supplies the selected set of pixels tothe estimated error calculating unit 412-1, and the pixel selecting unit411-2 supplies the selected set of pixels to the estimated errorcalculating unit 412-2. In the same way, each pixel selecting unit 411-3through pixel selecting unit 411-L supplies the selected set of pixelsto each estimated error calculating unit 412-3 through estimated errorcalculating unit 412-L.

The estimated error calculating unit 412-1 through estimated errorcalculating unit 412-L detect the correlation of the pixels values ofthe pixels at positions in the multiple sets, supplied from each of thepixel selecting unit 411-1 through pixel selecting unit 411-L. Forexample, the estimated error calculating unit 412-1 through estimatederror calculating unit 412-L calculates, as a value indicating thecorrelation, the sum of absolute values of difference between the pixelvalues of the pixels of the set containing the pixel of interest, andthe pixel values of the pixels at corresponding positions in other sets,supplied from one of the pixel selecting unit 411-1 through pixelselecting unit 411-L.

More specifically, based on the pixel values of the pixels of the setcontaining the pixel of interest and the pixel values of the pixels ofthe set made up of pixels belonging to one vertical row of pixels to theleft side of the pixel of interest supplied from one of the pixelselecting unit 411-1 through pixel selecting unit 411-L, the estimatederror calculating unit 412-1 through estimated error calculating unit412-L calculates the difference of the pixel values of the topmostpixel, then calculates the difference of the pixel values of the secondpixel from the top, and so on to calculate the absolute values ofdifference of the pixel values in order from the top pixel, and furthercalculates the sum of absolute values of the calculated differences.Based on the pixel values of the pixels of the set containing the pixelof interest and the pixel values of the pixels of the set made up ofpixels belonging to one vertical row of pixels two to the left from thepixel of interest supplied from one of the pixel selecting unit 411-1through pixel selecting unit 411-L, the estimated error calculating unit412-1 through estimated error calculating unit 412-L calculates theabsolute values of difference of the pixel values in order from the toppixel, and calculates the sum of absolute values of the calculateddifferences.

Then, based on the pixel values of the pixels of the set containing thepixel of interest and the pixel values of the pixels of the set made upof pixels belonging to one vertical row of pixels to the right side ofthe pixel of interest supplied from one of the pixel selecting unit411-1 through pixel selecting unit 411-L, the estimated errorcalculating unit 412-1 through estimated error calculating unit 412-Lcalculates the difference of the pixel values of the topmost pixel, thencalculates the difference of the pixel values of the second pixel fromthe top, and so on to calculate the absolute values of difference of thepixel values in order from the top pixel, and further calculates the sumof absolute values of the calculated differences. Based on the pixelvalues of the pixels of the set containing the pixel of interest and thepixel values of the pixels of the set made up of pixels belonging to onevertical row of pixels two to the right from the pixel of interestsupplied from one of the pixel selecting unit 411-1 through pixelselecting unit 411-L, the estimated error calculating unit 412-1 throughestimated error calculating unit 412-L calculates the absolute values ofdifference of the pixel values in order from the top pixel, andcalculates the sum of absolute values of the calculated differences.

The estimated error calculating unit 412-1 through estimated errorcalculating unit 412-L add all of the sums of absolute values ofdifference of the pixel values thus calculated, thereby calculating theaggregate of absolute values of difference of the pixel values.

The estimated error calculating unit 412-1 through estimated errorcalculating unit 412-L supply information indicating the detectedcorrelation to the smallest error angle selecting unit 413. For example,the estimated error calculating unit 412-1 through estimated errorcalculating unit 412-L supply the aggregate of absolute values ofdifference of the pixel values calculated, to the smallest error angleselecting unit 413.

Note that the estimated error calculating unit 412-1 through estimatederror calculating unit 412-L are not restricted to the sum of absolutevalues of difference of pixel values, and can also calculate othervalues as correlation values as well, such as the sum of squareddifferences of pixel values, or correlation coefficients based on pixelvalues, and so forth.

The smallest error angle selecting unit 413 detects the data continuityangle based on the reference axis in the input image which correspondsto the continuity of the image which is the lost actual world 1 lightsignals, based on the correlation detected by the estimated errorcalculating unit 412-1 through estimated error calculating unit 412-Lwith regard to mutually different angles. That is to say, based on thecorrelation detected by the estimated error calculating unit 412-1through estimated error calculating unit 412-L with regard to mutuallydifferent angles, the smallest error angle selecting unit 413 selectsthe greatest correlation, and takes the angle regarding which theselected correlation was detected as the data continuity angle based onthe reference axis, thereby detecting the data continuity angle based onthe reference axis in the input image.

For example, of the aggregates of absolute values of difference of thepixel values supplied from the estimated error calculating unit 412-1through estimated error calculating unit 412-L, the smallest error angleselecting unit 413 selects the smallest aggregate. With regard to thepixel set of which the selected aggregate was calculated, the smallesterror angle selecting unit 413 makes reference to a pixel belonging tothe one vertical row of pixels two to the left from the pixel ofinterest and at the closest pixel position to the straight line, and toa pixel belonging to the one vertical row of pixels two to the rightfrom the pixel of interest and at the closest pixel position to thestraight line.

As shown in FIG. 54, the smallest error angle selecting unit 413 obtainsthe distance S in the vertical direction of the position of the pixelsto reference from the position of the pixel of interest. As shown inFIG. 55, the smallest error angle selecting unit 413 detects the angle θof data continuity based on the axis indicating the spatial direction Xwhich is the reference axis in the input image which is image data, thatcorresponds to the lost actual world 1 light signals continuity, fromExpression (37).

$\begin{matrix}{\theta = {\tan^{- 1}\frac{S}{2}}} & (37)\end{matrix}$

Next, description will be made regarding the processing of the pixelselecting unit 411-1 through pixel selecting unit 411-L in the eventthat the data continuity angle indicated by the activity information isa value of any from 0 degrees to 45 degrees and 135 degrees to 180degrees.

The pixel selecting unit 411-1 through pixel selecting unit 411-L setstraight lines of predetermined angles which pass through the pixel ofinterest, with the axis indicating the spatial direction X as thereference axis, and select, of the pixels belonging to a horizontal rowof pixels to which the pixel of interest belongs, a predetermined numberof pixels to the left of the pixel of interest, and predetermined numberof pixels to the right of the pixel of interest, and the pixel ofinterest, as a pixel set.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a horizontal row of pixels above thehorizontal row of pixels to which the pixel of interest belongs, a pixelat the position closest to the straight line set for each. The pixelselecting unit 411-1 through pixel selecting unit 411-L then select,from the pixels belonging to the horizontal row of pixels above thehorizontal row of pixels to which the pixel of interest belongs, apredetermined number of pixels to the left of the selected pixel, apredetermined number of pixels to the right of the selected pixel, andthe selected pixel, as a pixel set.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a horizontal row of pixels two abovethe horizontal row of pixels to which the pixel of interest belongs, apixel at the position closest to the straight line set for each. Thepixel selecting unit 411-1 through pixel selecting unit 411-L thenselect, from the pixels belonging to the horizontal row of pixels twoabove the horizontal row of pixels to which the pixel of interestbelongs, a predetermined number of pixels to the left of the selectedpixel, a predetermined number of pixels to the right of the selectedpixel, and the selected pixel, as a pixel set.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a horizontal row of pixels below thehorizontal row of pixels to which the pixel of interest belongs, a pixelat the position closest to the straight line set for each. The pixelselecting unit 411-1 through pixel selecting unit 411-L then select,from the pixels belonging to the horizontal row of pixels below thehorizontal row of pixels to which the pixel of interest belongs, apredetermined number of pixels to the left of the selected pixel, apredetermined number of pixels to the right of the selected pixel, andthe selected pixel, as a pixel set.

The pixel selecting unit 411-1 through pixel selecting unit 411-Lselect, from pixels belonging to a horizontal row of pixels two belowthe horizontal row of pixels to which the pixel of interest belongs, apixel at the position closest to the straight line set for each. Thepixel selecting unit 411-1 through pixel selecting unit 411-L thenselect, from the pixels belonging to the horizontal row of pixels twobelow the horizontal row of pixels to which the pixel of interestbelongs, a predetermined number of pixels to the left of the selectedpixel, a predetermined number of pixels to the right of the selectedpixel, and the selected pixel, as a pixel set.

Thus, the pixel selecting unit 411-1 through pixel selecting unit 411-Leach select five sets of pixels.

The pixel selecting unit 411-1 through pixel selecting unit 411-L selectpixel sets for mutually different angles. For example, the pixelselecting unit 411-1 selects sets of pixels regarding 0 degrees, thepixel selecting unit 411-2 selects sets of pixels regarding 2.5 degrees,and the pixel selecting unit 411-3 selects sets of pixels regarding 5degrees. The pixel selecting unit 411-1 through pixel selecting unit411-L select sets of pixels regarding angles every 2.5 degrees, from 7.5degrees through 45 degrees and from 135 degrees through 180 degrees.

The pixel selecting unit 411-1 supplies the selected set of pixels tothe estimated error calculating unit 412-1, and the pixel selecting unit411-2 supplies the selected set of pixels to the estimated errorcalculating unit 412-2. In the same way, each pixel selecting unit 411-3through pixel selecting unit 411-L supplies the selected set of pixelsto each estimated error calculating unit 412-3 through estimated errorcalculating unit 412-L.

The estimated error calculating unit 412-1 through estimated errorcalculating unit 412-L detect the correlation of the pixels values ofthe pixels at positions in the multiple sets, supplied from each of thepixel selecting unit 411-1 through pixel selecting unit 411-L. Theestimated error calculating unit 412-1 through estimated errorcalculating unit 412-L supply information indicating the detectedcorrelation to the smallest error angle selecting unit 413.

The smallest error angle selecting unit 413 detects the data continuityangle based on the reference axis in the input image which correspondsto the continuity of the image which is the lost actual world 1 lightsignals, based on the correlation detected by the estimated errorcalculating unit 412-1 through estimated error calculating unit 412-L.

Next, data continuity detection processing with the data continuitydetecting unit 101 of which the configuration is shown in FIG. 49,corresponding to the processing in step S101, will be described withreference to the flowchart in FIG. 56.

In step S401, the activity detecting unit 401 and the data selectingunit 402 select the pixel of interest which is a pixel of interest fromthe input image. The activity detecting unit 401 and the data selectingunit 402 select the same pixel of interest. For example, the activitydetecting unit 401 and the data selecting unit 402 select the pixel ofinterest from the input image in raster scan order.

In step S402, the activity detecting unit 401 detects activity withregard to the pixel of interest. For example, the activity detectingunit 401 detects activity based on the difference of pixel values ofpixels aligned in the vertical direction of a block made up of apredetermined number of pixels centered on the pixel of interest, andthe difference of pixel values of pixels aligned in the horizontaldirection.

The activity detecting unit 401 detects activity in the spatialdirection as to the pixel of interest, and supplies activity informationindicating the detected results to the data selecting unit 402 and thecontinuity direction derivation unit 404.

In step S403, the data selecting unit 402 selects, from a row of pixelsincluding the pixel of interest, a predetermined number of pixelscentered on the pixel of interest, as a pixel set. For example, the dataselecting unit 402 selects a predetermined number of pixels above or tothe left of the pixel of interest, and a predetermined number of pixelsbelow or to the right of the pixel of interest, which are pixelsbelonging to a vertical or horizontal row of pixels to which the pixelof interest belongs, and also the pixel of interest, as a pixel set.

In step S404, the data selecting unit 402 selects, as a pixel set, apredetermined number of pixels each from a predetermined number of pixelrows for each angle in a predetermined range based on the activitydetected by the processing in step S402. For example, the data selectingunit 402 sets straight lines with angles of a predetermined range whichpass through the pixel of interest, with the axis indicating the spatialdirection X as the reference axis, selects a pixel which is one or tworows away from the pixel of interest in the horizontal direction orvertical direction and which is closest to the straight line, andselects a predetermined number of pixels above or to the left of theselected pixel, and a predetermined number of pixels below or to theright of the selected pixel, and the selected pixel closest to the line,as a pixel set. The data selecting unit 402 selects pixel sets for eachangle.

The data selecting unit 402 supplies the selected pixel sets to theerror estimating unit 403.

In step S405, the error estimating unit 403 calculates the correlationbetween the set of pixels centered on the pixel of interest, and thepixel sets selected for each angle. For example, the error estimatingunit 403 calculates the sum of absolute values of difference of thepixel values of the pixels of the set including the pixel of interestand the pixel values of the pixels at corresponding positions in othersets, for each angle.

The angle of data continuity may be detected based on the correlationbetween pixel sets selected for each angle.

The error estimating unit 403 supplies the information indicating thecalculated correlation to the continuity direction derivation unit 404.

In step S406, from position of the pixel set having the strongestcorrelation based on the correlation calculated in the processing instep S405, the continuity direction derivation unit 404 detects the datacontinuity angle based on the reference axis in the input image which isimage data that corresponds to the lost actual world 1 light signalcontinuity. For example, the continuity direction derivation unit 404selects the smallest aggregate of the aggregate of absolute values ofdifference of pixel values, and detects the data continuity angle θ fromthe position of the pixel set regarding which the selected aggregate hasbeen calculated.

The continuity direction derivation unit 404 outputs data continuityinformation indicating the angle of the data continuity that has beendetected.

In step S407, the data selecting unit 402 determines whether or notprocessing of all pixels has ended, and in the event that determinationis made that processing of all pixels has not ended, the flow returns tostep S401, a pixel of interest is selected from pixels not yet taken asthe pixel of interest, and the above-described processing is repeated.

In the event that determination is made in step S407 that processing ofall pixels has ended, the processing ends.

Thus, the data continuity detecting unit 101 can detect the datacontinuity angle based on the reference axis in the image data,corresponding to the lost actual world 1 light signal continuity.

Note that an arrangement may be made wherein the data continuitydetecting unit 101 of which the configuration is shown in FIG. 49detects activity in the spatial direction of the input image with regardto the pixel of interest which is a pixel of interest in the frame ofinterest which is a frame of interest, extracts multiple pixel sets madeup of a predetermined number of pixels in one row in the verticaldirection or one row in the horizontal direction from the frame ofinterest and from each of frames before or after time-wise the frame ofinterest, for each angle and movement vector based on the pixel ofinterest and the space-directional reference axis, according to thedetected activity, detects the correlation of the extracted pixel sets,and detects the data continuity angle in the time direction and spatialdirection in the input image, based on this correlation.

For example, as shown in FIG. 57, the data selecting unit 402 extractsmultiple pixel sets made up of a predetermined number of pixels in onerow in the vertical direction or one row in the horizontal directionfrom frame #n which is the frame of interest, frame #n−1, and frame#n+1, for each angle and movement vector based on the pixel of interestand the space-directional reference axis, according to the detectedactivity.

The frame #n−1 is a frame which is previous to the frame #n time-wise,and the frame #n+1 is a frame following the frame #n time-wise. That isto say, the frame #n−1, frame #n, and frame #n+1, are displayed in theorder of frame #n−1, frame #n, and frame #n+1.

The error estimating unit 403 detects the correlation of pixel sets foreach single angle and single movement vector, with regard to themultiple sets of the pixels that have been extracted. The continuitydirection derivation unit 404 detects the data continuity angle in thetemporal direction and spatial direction in the input image whichcorresponds to the lost actual world 1 light signal continuity, based onthe correlation of pixel sets, and outputs the data continuityinformation indicating the angle.

Next, description will be made regarding another embodiment example ofthe actual world estimating unit 102 (FIG. 3) with reference to FIG. 58through FIG. 88.

FIG. 58 is a diagram for describing the principle of this embodimentexample.

As shown in FIG. 58, a signal (light intensity allocation) in the actualworld 1, which is an image cast on the sensor 2, is represented with apredetermined function F. Note that hereafter, with the description ofthis embodiment example, the signal serving as an image in the actualworld 1 is particularly referred to as a light signal, and the functionF is particularly referred to as a light signal function F.

With this embodiment example, in the event that the light signal in theactual world 1 represented with the light signal function F haspredetermined continuity, the actual world estimating unit 102 estimatesthe light signal function F by approximating the light signal function Fwith a predetermined function f using an input image (image dataincluding continuity of data corresponding to continuity) from thesensor 2, and data continuity information (data continuity informationcorresponding to continuity of the input image data) from the datacontinuity detecting unit 101. Note that with the description of thisembodiment example, the function f is particularly referred to as anapproximation function f, hereafter.

In other words, with this embodiment example, the actual worldestimating unit 102 approximates (describes) the image (light signal inthe actual world 1) represented with the light signal function F using amodel 161 (FIG. 4) represented with the approximation function f.Accordingly, hereafter, this embodiment example is referred to as afunction approximating method.

Now, description will be made regarding the background wherein thepresent applicant has invented the function approximating method, priorto entering the specific description of the function approximatingmethod.

FIG. 59 is a diagram for describing integration effects in the case inwhich the sensor 2 is treated as a CCD.

As shown in FIG. 59, multiple detecting elements 2-1 are disposed on theplane of the sensor 2.

With the example in FIG. 59, a direction in parallel with apredetermined side of the detecting elements 2-1 is taken as the Xdirection, which is one direction in the spatial direction, and the adirection orthogonal to the X direction is taken as the Y direction,which is another direction in the spatial direction. Also, the directionperpendicular to the X-Y plane is taken as the direction t serving asthe temporal direction.

Also, with the example in FIG. 59, the spatial shape of each detectingelement 2-1 of the sensor 2 is represented with a square of which oneside is 1 in length. The shutter time (exposure time) of the sensor 2 isrepresented with 1.

Further, with the example in FIG. 59, the center of one detectingelement 2-1 of the sensor 2 is taken as the origin (position x=0 in theX direction, and position y=0 in the Y direction) in the spatialdirection (X direction and Y direction), and the intermediatepoint-in-time of the exposure time is taken as the origin (position t=0in the t direction) in the temporal direction (t direction).

In this case, the detecting element 2-1 of which the center is in theorigin (x=0, y=0) in the spatial direction subjects the light signalfunction F(x, y, t) to integration with a range between −0.5 and 0.5 inthe X direction, range between −0.5 and 0.5 in the Y direction, andrange between −0.5 and 0.5 in the t direction, and outputs the integralvalue thereof as a pixel value P.

That is to say, the pixel value P output from the detecting element 2-1of which the center is in the origin in the spatial direction isrepresented with the following Expression (38).

$\begin{matrix}{P = {\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{{F\left( {x,y,t} \right)}\ {\mathbb{d}x}\ {\mathbb{d}y}\ {\mathbb{d}t}}}}}} & (38)\end{matrix}$

The other detecting elements 2-1 also output the pixel value P shown inExpression (38) by taking the center of the subject detecting element2-1 as the origin in the spatial direction in the same way.

FIG. 60 is a diagram for describing a specific example of theintegration effects of the sensor 2.

In FIG. 60, the X direction and Y direction represent the X directionand Y direction of the sensor 2 (FIG. 59).

A portion 2301 of the light signal in the actual world 1 (hereafter,such a portion is referred to as a region) represents an example of aregion having predetermined continuity.

Note that the region 2301 is a portion of the continuous light signal(continuous region). On the other hand, in FIG. 60, the region 2301 isshown as divided into 20 small regions (square regions) in reality. Thisis because of representing that the size of the region 2301 isequivalent to the size wherein the four detecting elements (pixels) ofthe sensor 2 in the X direction, and also the five detecting elements(pixels) of the sensor 2 in the Y direction are arrayed. That is to say,each of the 20 small regions (virtual regions) within the region 2301 isequivalent to one pixel.

Also, a white portion shown in the drawing within the region 2301represents a light signal corresponding to a fine line. Accordingly, theregion 2301 has continuity in the direction wherein a fine linecontinues. Hereafter, the region 2301 is referred to as thefine-line-including actual world region 2301.

In this case, when the fine-line-including actual world region 2301 (aportion of a light signal in the actual world 1) is detected by thesensor 2, region 2302 (hereafter, this is referred to as afine-line-including data region 2302) of the input image (pixel values)is output from the sensor 2 by integration effects.

Note that each pixel of the fine-line-including data region 2302 isrepresented as an image in the drawing, but is data representing apredetermined value in reality. That is to say, the fine-line-includingactual world region 2301 is changed (distorted) to thefine-line-including data region 2302, which is divided into 20 pixels(20 pixels in total of 4 pixels in the X direction and also 5 pixels inthe Y direction) each having a predetermined pixel value by theintegration effects of the sensor 2.

FIG. 61 is a diagram for describing another specific example (exampledifferent from FIG. 60) of the integration effects of the sensor 2.

In FIG. 61, the X direction and Y direction represent the X directionand Y direction of the sensor 2 (FIG. 59).

A portion (region) 2303 of the light signal in the actual world 1represents another example (example different from thefine-line-including actual world region 2301 in FIG. 60) of a regionhaving predetermined continuity.

Note that the region 2303 is a region having the same size as thefine-line-including actual world region 2301. That is to say, the region2303 is also a portion of the continuous light signal in the actualworld 1 (continuous region) as with the fine-line-including actual worldregion 2301 in reality, but is shown as divided into 20 small regions(square regions) equivalent to one pixel of the sensor 2 in FIG. 61.

Also, the region 2303 includes a first portion having predeterminedfirst light intensity (value), and a second portion edge havingpredetermined second light intensity (value). Accordingly, the region2303 has continuity in the direction wherein the edges continue.Hereafter, the region 2303 is referred to as thetwo-valued-edge-including actual world region 2303.

In this case, when the two-valued-edge-including actual world region2303 (a portion of the light signal in the actual world 1) is detectedby the sensor 2, a region 2304 (hereafter, referred to astwo-valued-edge-including data region 2304) of the input image (pixelvalue) is output from the sensor 2 by integration effects.

Note that each pixel value of the two-valued-edge-including data region2304 is represented as an image in the drawing as with thefine-line-including data region 2302, but is data representing apredetermined value in reality. That is to say, thetwo-valued-edge-including actual world region 2303 is changed(distorted) to the two-valued-edge-including data region 2304, which isdivided into 20 pixels (20 pixels in total of 4 pixels in the Xdirection and also 5 pixels in the Y direction) each having apredetermined pixel value by the integration effects of the sensor 2.

Conventional image processing devices have regarded image data outputfrom the sensor 2 such as the fine-line-including data region 2302,two-valued-edge-including data region 2304, and the like as the origin(basis), and also have subjected the image data to the subsequent imageprocessing. That is to say, regardless of that the image data outputfrom the sensor 2 had been changed (distorted) to data different fromthe light signal in the actual world 1 by integration effects, theconventional image processing devices have performed image processing onassumption that the data different from the light signal in the actualworld 1 is correct.

As a result, the conventional image processing devices have provided aproblem wherein based on the waveform (image data) of which the detailsin the actual world is distorted at the stage wherein the image data isoutput from the sensor 2, it is very difficult to restore the originaldetails from the waveform.

Accordingly, with the function approximating method, in order to solvethis problem, as described above (as shown in FIG. 58), the actual worldestimating unit 102 estimates the light signal function F byapproximating the light signal function F (light signal in the actualworld 1) with the approximation function f based on the image data(input image) such as the fine-line-including data region 2302, andtwo-valued-edge-including data region 2304 output from the sensor 2.

Thus, at a later stage than the actual world estimating unit 102 (inthis case, the image generating unit 103 in FIG. 3), the processing canbe performed by taking the image data wherein integration effects aretaken into consideration, i.e., image data that can be represented withthe approximation function f as the origin.

Hereafter, description will be made independently regarding threespecific methods (first through third function approximating methods),of such a function approximating method with reference to the drawings.

First, description will be made regarding the first functionapproximating method with reference to FIG. 62 through FIG. 76.

FIG. 62 is a diagram representing the fine-line-including actual worldregion 2301 shown in FIG. 60 described above again.

In FIG. 62, the X direction and Y direction represent the X directionand Y direction of the sensor 2 (FIG. 59).

The first function approximating method is a method for approximating aone-dimensional waveform (hereafter, such a waveform is referred to asan X cross-sectional waveform F(x)) wherein the light signal functionF(x, y, t) corresponding to the fine-line-including actual world region2301 such as shown in FIG. 62 is projected in the X direction (directionof an arrow 2311 in the drawing), with the approximation function f(x)serving as an n-dimensional (n is an arbitrary integer) polynomial, forexample. Accordingly, hereafter, the first function approximating methodis particularly referred to as a one-dimensional approximating method.

Note that with the one-dimensional approximating method, the Xcross-sectional waveform F(x), which is to be approximated, is notrestricted to a waveform corresponding to the fine-line-including actualworld region 2301 in FIG. 62, of course. That is to say, as describedlater, with the one-dimensional approximating method, any waveform canbe approximated as long as the X cross-sectional waveform F(x)corresponds to the light signals in the actual world 1 havingcontinuity.

Also, the direction of the projection of the light signal function F(x,y, t) is not restricted to the X direction, or rather the Y direction ort direction may be employed. That is to say, with the one-dimensionalapproximating method, a function F(y) wherein the light signal functionF(x, y, t) is projected in the Y direction may be approximated with apredetermined approximation function f(y), or a function F(t) whereinthe light signal function F(x, y, t) is projected in the t direction maybe approximated with a predetermined approximation function f(t).

More specifically, the one-dimensional approximating method is a methodfor approximating, for example, the X cross-sectional waveform F(x) withthe approximation function f(x) serving as an n-dimensional polynomialsuch as shown in the following Expression (39).

$\begin{matrix}{{f(x)} = {{w_{0} + {w_{1}x} + {w_{2}x} + \ldots + {w_{n}x^{n}}} = {\sum\limits_{i = 0}^{n}{w_{i}x^{\; i}}}}} & (39)\end{matrix}$

That is to say, with the one-dimensional approximating method, theactual world estimating unit 102 estimates the X cross-sectionalwaveform F(x) by calculating the coefficient (features) w_(i) of x^(i)in Expression (39).

This calculation method of the features w_(i) is not restricted to aparticular method, for example, the following first through thirdmethods may be employed.

That is to say, the first method is a method that has been employed sofar.

On the other hand, the second method is a method that has been newlyinvented by the present applicant, which is a method that considerscontinuity in the spatial direction as to the first method.

However, as described later, with the first and second methods, theintegration effects of the sensor 2 are not taken into consideration.Accordingly, an approximation function f(x) obtained by substituting thefeatures w_(i) calculated by the first method or the second method forthe above Expression (39) is an approximation function regarding aninput image, but strictly speaking, cannot be referred to as theapproximation function of the X cross-sectional waveform F(x).

Consequently, the present applicant has invented the third method thatcalculates the features w_(i) further in light of the integrationeffects of the sensor 2 as to the second method. An approximationfunction f(x) obtained by substituting the features w_(i) calculatedwith this third method for the above Expression (39) can be referred toas the approximation function of the X cross-sectional waveform F(x) inthat the integration effects of the sensor 2 are taken intoconsideration.

Thus, strictly speaking, the first method and the second method cannotbe referred to as the one-dimensional approximating method, and thethird method alone can be referred to as the one-dimensionalapproximating method.

In other words, as shown in FIG. 63, the second method is different fromthe one-dimensional approximating method. That is to say, FIG. 63 is adiagram for describing the principle of the embodiment corresponding tothe second method.

As shown in FIG. 63, with the embodiment corresponding to the secondmethod, in the event that the light signal in the actual world 1represented with the light signal function F has predeterminedcontinuity, the actual world estimating unit 102 does not approximatethe X cross-sectional waveform F(x) with an input image (image dataincluding continuity of data corresponding to continuity) from thesensor 2, and data continuity information (data continuity informationcorresponding to continuity of input image data) from the datacontinuity detecting unit 101, but approximates the input image from thesensor 2 with a predetermined approximation function f₂(x).

Thus, it is hard to say that the second method is a method having thesame level as the third method in that approximation of the input imagealone is performed without considering the integral effects of thesensor 2. However, the second method is a method superior to theconventional first method in that the second method takes continuity inthe spatial direction into consideration.

Hereafter, description will be made independently regarding the detailsof the first method, second method, and third method in this order.

Note that hereafter, in the event that the respective approximationfunctions f(x) generated by the first method, second method, and thirdmethod are distinguished from that of the other method, they areparticularly referred to as approximation function f₁(x), approximationfunction f₂(x), and approximation function f₃(x) respectively.

First, description will be made regarding the details of the firstmethod.

With the first method, on condition that the approximation functionf₁(x) shown in the above Expression (39) holds within thefine-line-including actual world region 2301 in FIG. 64, the followingprediction equation (40) is defined.

$\begin{matrix}{{P\left( {x,y} \right)} = {{f_{1}(x)} + e}} & (40)\end{matrix}$

In Expression (40), x represents a pixel position relative as to the Xdirection from a pixel of interest. y represents a pixel positionrelative as to the Y direction from the pixel of interest. e representsa margin of error. Specifically, for example, as shown in FIG. 64, letus say that the pixel of interest is the second pixel in the X directionfrom the left, and also the third pixel in the Y direction from thebottom in the drawing, of the fine-line-including data region 2302 (dataof which the fine-line-including actual world region 2301 (FIG. 62) isdetected by the sensor 2, and output). Also, let us say that the centerof the pixel of interest is the origin (0, 0), and a coordinates system(hereafter, referred to as a pixel-of-interest coordinates system) ofwhich axes are an x axis and y axis respectively in parallel with the Xdirection and Y direction of the sensor 2 (FIG. 59) is set. In thiscase, the coordinates value (x, y) of the pixel-of-interest coordinatessystem represents a relative pixel position.

Also, in Expression (40), P(x, y) represents a pixel value in therelative pixel positions (x, y). Specifically, in this case, the P(x, y)within the fine-line-including data region 2302 is such as shown in FIG.65.

FIG. 65 represents this pixel value P(x, y) in a graphic manner.

In FIG. 65, the respective vertical axes of the graphs represent pixelvalues, and the horizontal axes represent a relative position x in the Xdirection from the pixel of interest. Also, in the drawing, the dashedline in the first graph from the top represents an input pixel valueP(x, −2), the broken triple-dashed line in the second graph from the toprepresents an input pixel value P(x, −1), the solid line in the thirdgraph from the top represents an input pixel value P(x, 0), the brokenline in the fourth graph from the top represents an input pixel valueP(x, 1), and the broken double-dashed line in the fifth graph from thetop (the first from the bottom) represents an input pixel value P(x, 2)respectively.

Upon the 20 input pixel values P(x, −2), P(x, −1), P (x, 0), P(x, 1),and P(x, 2) (however, x is any one integer value of −1 through 2) shownin FIG. 65 being substituted for the above Expression (40) respectively,20 equations as shown in the following Expression (41) are generated.Note that each e_(k)(k is any one of integer values 1 through 20)represents a margin of error.

$\begin{matrix}{{{P\left( {{- 1},{- 2}} \right)} = {{f_{1}\left( {- 1} \right)} + e_{1}}}{{P\left( {0,{- 2}} \right)} = {{f_{1}(0)} + e_{2}}}{{P\left( {1,{- 2}} \right)} = {{f_{1}(1)} + e_{3}}}{{P\left( {2,{- 2}} \right)} = {{f_{1}(2)} + e_{4}}}{{P\left( {{- 1},{- 1}} \right)} = {{f_{1}\left( {- 1} \right)} + e_{5}}}{{P\left( {0,{- 1}} \right)} = {{f_{1}(0)} + e_{6}}}{{P\left( {1,{- 1}} \right)} = {{f_{1}(1)} + e_{7}}}{{P\left( {2,{- 1}} \right)} = {{f_{1}(2)} + e_{8}}}{{P\left( {{- 1},0} \right)} = {{f_{1}\left( {- 1} \right)} + e_{9}}}{{P\left( {0,0} \right)} = {{f_{1}(0)} + e_{10}}}{{P\left( {1,0} \right)} = {{f_{1}(1)} + e_{11}}}{{P\left( {2,0} \right)} = {{f_{1}(2)} + e_{12}}}{{P\left( {{- 1},1} \right)} = {{f_{1}\left( {- 1} \right)} + e_{13}}}{{P\left( {0,1} \right)} = {{f_{1}(0)} + e_{14}}}{{P\left( {1,1} \right)} = {{f_{1}(1)} + e_{15}}}{{P\left( {2,1} \right)} = {{f_{1}(2)} + e_{16}}}{{P\left( {{- 1},2} \right)} = {{f_{1}\left( {- 1} \right)} + e_{17}}}{{P\left( {0,2} \right)} = {{f_{1}(0)} + e_{18}}}{{P\left( {1,2} \right)} = {{f_{1}(1)} + e_{19}}}{{P\left( {2,2} \right)} = {{f_{1}(2)} + e_{20}}}} & (41)\end{matrix}$

Expression (41) is made up of 20 equations, so in the event that thenumber of the features w_(i) of the approximation function f₁(x) is lessthan 20, i.e., in the event that the approximation function f₁(x) is apolynomial having the number of dimensions less than 19, the featuresw_(i) can be calculated using the least square method, for example. Notethat the specific solution of the least square method will be describedlater.

For example, if we say that the number of dimensions of theapproximation function f₁(x) is five, the approximation function f₁(x)calculated with the least square method using Expression (41) (theapproximation function f₁(x) generated by the calculated features w_(i))becomes a curve shown in FIG. 66.

Note that in FIG. 66, the vertical axis represents pixel values, and thehorizontal axis represents a relative position x from the pixel ofinterest.

That is to say, for example, if we supplement the respective 20 pixelvalues P(x, y) (the respective input pixel values P(x, −2), P(x, −1),P(x, 0), P(x, 1), and P(x, 2) shown in FIG. 65) making up thefine-line-including data region 2302 in FIG. 64 along the x axis withoutany modification (if we regard a relative position y in the Y directionas constant, and overlay the five graphs shown in FIG. 65), multiplelines (dashed line, broken triple-dashed line, solid line, broken line,and broken double-dashed line) in parallel with the x axis, such asshown in FIG. 66, are distributed.

However, in FIG. 66, the dashed line represents the input pixel valueP(x, −2), the broken triple-dashed line represents the input pixel valueP(x, −1), the solid line represents the input pixel value P(x, 0), thebroken line represents the input pixel value P(x, 1), and the brokendouble-dashed line represents the input pixel value P(x, 2)respectively. Also, in the event of the same pixel value, lines morethan 2 lines are overlaid in reality, but in FIG. 66, the lines aredrawn so as to distinguish each line, and so as not to overlay eachline.

The respective 20 input pixel values (P(x, −2), P(x, −1), P(x, 0), P(x,1), and P(x, 2)) thus distributed, and a regression curve (theapproximation function f₁(x) obtained by substituting the features w_(i)calculated with the least square method for the above Expression (38))so as to minimize the error of the value f₁(x) become a curve(approximation function f₁(x)) shown in FIG. 66.

Thus, the approximation function f₁(x) represents nothing but a curveconnecting in the X direction the means of the pixel values (pixelvalues having the same relative position x in the X direction from thepixel of interest) P(x, −2), P(x, −1), P(x, 0), P(x, 1), and P(x, 2) inthe Y direction. That is to say, the approximation function f₁(x) isgenerated without considering continuity in the spatial directionincluded in the light signal.

For example, in this case, the fine-line-including actual world region2301 (FIG. 62) is regarded as a subject to be approximated. Thisfine-line-including actual world region 2301 has continuity in thespatial direction, which is represented with a gradient G_(F), such asshown in FIG. 67. Note that in FIG. 67, the X direction and Y directionrepresent the X direction and Y direction of the sensor 2 (FIG. 59).

Accordingly, the data continuity detecting unit 101 (FIG. 58) can outputan angle θ (angle θ generated between the direction of data continuityrepresented with a gradient G_(f) corresponding to the gradient G_(F),and the X direction) such as shown in FIG. 67 as data continuityinformation corresponding to the gradient G_(F) as continuity in thespatial direction

However, with the first method, the data continuity information outputfrom the data continuity detecting unit 101 is not employed at all.

In other words, such as shown in FIG. 67, the direction of continuity inthe spatial direction of the fine-line-including actual world region2301 is a general angle θ direction. However, the first method is amethod for calculating the features w_(i) of the approximation functionf₁(x) on assumption that the direction of continuity in the spatialdirection of the fine-line-including actual world region 2301 is the Ydirection (i.e., on assumption that the angle θ is 90°).

Consequently, the approximation function f₁(x) becomes a function ofwhich the waveform gets dull, and the detail decreases than the originalpixel value. In other words, though not shown in the drawing, with theapproximation function f₁(x) generated with the first method, thewaveform thereof becomes a waveform different from the actual Xcross-sectional waveform F(x).

To this end, the present applicant has invented the second method forcalculating the features w_(i) by further taking continuity in thespatial direction into consideration (utilizing the angle θ) as to thefirst method.

That is to say, the second method is a method for calculating thefeatures w_(i) of the approximation function f₂(x) on assumption thatthe direction of continuity of the fine-line-including actual worldregion 2301 is a general angle θ direction.

Specifically, for example, the gradient G_(f) representing continuity ofdata corresponding to continuity in the spatial direction is representedwith the following Expression (42).

$\begin{matrix}{G_{f} = {{\tan\;\theta} = \frac{\mathbb{d}y}{\mathbb{d}x}}} & (42)\end{matrix}$

Note that in Expression (42), dx represents the amount of fine movementin the X direction such as shown in FIG. 67, dy represents the amount offine movement in the Y direction as to the dx such as shown in FIG. 67.

In this case, if we define the shift amount C_(x)(y) as shown in thefollowing Expression (43), with the second method, an equationcorresponding to Expression (40) employed in the first method becomessuch as the following Expression (44).

$\begin{matrix}{{C_{x}(y)} = \frac{y}{G_{f}}} & (43) \\{{P\left( {x,y} \right)} = {{f_{2}\left( {x - {C_{x}(y)}} \right)} + e}} & (44)\end{matrix}$

That is to say, Expression (40) employed in the first method representsthat the position x in the X direction of the pixel center position (x,y) is the same value regarding the pixel value P(x, y) of any pixelpositioned in the same position. In other words, Expression (40)represents that pixels having the same pixel value continue in the Ydirection (exhibits continuity in the Y direction).

On the other hand, Expression (44) employed in the second methodrepresents that the pixel value P(x, y) of a pixel of which the centerposition is (x, y) is not identical to the pixel value (approximateequivalent to f₂(x)) of a pixel positioned in a place distant from thepixel of interest (a pixel of which the center position is the origin(0, 0)) in the X direction by x, and is the same value as the pixelvalue (approximate equivalent to f₂(x+C_(x)(y)) of a pixel positioned ina place further distant from the pixel thereof in the X direction by theshift amount C_(x)(y) (pixel positioned in a place distant from thepixel of interest in the X direction by x+C_(x)(y)). In other words,Expression (44) represents that pixels having the same pixel valuecontinue in the angle θ direction corresponding to the shift amountC_(x)(y) (exhibits continuity in the general angle θ direction).

Thus, the shift amount C_(x)(y) is the amount of correction consideringcontinuity (in this case, continuity represented with the gradient G_(F)in FIG. 67 (strictly speaking, continuity of data represented with thegradient G_(f))) in the spatial direction, and Expression (44) isobtained by correcting Expression (40) with the shift amount C_(x)(y).

In this case, upon the 20 pixel values P(x, y) (however, x is any oneinteger value of −1 through 2, and y is any one integer value of −2through 2) of the fine-line-including data region 2302 shown in FIG. 64being substituted for the above Expression (44) respectively, 20equations as shown in the following Expression (45) are generated.

$\begin{matrix}{{{{P\left( {{- 1},{- 2}} \right)} = {{f_{2}\left( {{- 1} - {C_{x}\left( {- 2} \right)}} \right)} + e_{1}}}{{P\left( {0,{- 2}} \right)} = {{f_{2}\left( {0 - {C_{x}\left( {- 2} \right)}} \right)} + e_{2}}}{P\left( {1,{- 2}} \right)} = {{f_{2}\left( {1 - {C_{x}\left( {- 2} \right)}} \right)} + e_{3}}}{{P\left( {2,{- 2}} \right)} = {{f_{2}\left( {2 - {C_{x}\left( {- 2} \right)}} \right)} + e_{4}}}{{P\left( {{- 1},{- 1}} \right)} = {{f_{2}\left( {{- 1} - {C_{x}\left( {- 1} \right)}} \right)} + e_{5}}}{{P\left( {0,{- 1}} \right)} = {{f_{2}\left( {0 - {C_{x}\left( {- 1} \right)}} \right)} + e_{6}}}{{P\left( {1,{- 1}} \right)} = {{f_{2}\left( {1 - {C_{x}\left( {- 1} \right)}} \right)} + e_{7}}}{{P\left( {2,{- 1}} \right)} = {{f_{2}\left( {2 - {C_{x}\left( {- 1} \right)}} \right)} + e_{8}}}{{P\left( {{- 1},0} \right)} = {{f_{2}\left( {- 1} \right)} + e_{9}}}{{P\left( {0,0} \right)} = {{f_{2}(0)} + e_{10}}}{{P\left( {1,0} \right)} = {{f_{2}(1)} + e_{11}}}{{P\left( {2,0} \right)} = {{f_{2}(2)} + e_{12}}}{{P\left( {{- 1},1} \right)} = {{f_{2}\left( {{- 1} - {C_{x}(1)}} \right)} + e_{13}}}{{P\left( {0,1} \right)} = {{f_{2}\left( {0 - {C_{x}(1)}} \right)} + e_{14}}}{{P\left( {1,1} \right)} = {{f_{2}\left( {1 - {C_{x}(1)}} \right)} + e_{15}}}{{P\left( {2,1} \right)} = {{f_{2}\left( {2 - {C_{x}(1)}} \right)} + e_{16}}}{{P\left( {{- 1},2} \right)} = {{f_{2}\left( {{- 1} - {C_{x}(2)}} \right)} + e_{17}}}{{P\left( {0,2} \right)} = {{f_{2}\left( {0 - {C_{x}(2)}} \right)} + e_{18}}}{{P\left( {1,2} \right)} = {{f_{2}\left( {1 - {C_{x}(2)}} \right)} + e_{19}}}{{P\left( {2,2} \right)} = {{f_{2}\left( {2 - {C_{x}(2)}} \right)} + e_{20}}}} & (45)\end{matrix}$

Expression (45) is made up of 20 equations, as with the above Expression(41). Accordingly, with the second method, as with the first method, inthe event that the number of the features w_(i) of the approximationfunction f₂(x) is less than 20, i.e., the approximation function f₂(x)is a polynomial having the number of dimensions less than 19, thefeatures w_(i) can be calculated with the least square method, forexample. Note that the specific solution regarding the least squaremethod will be described later.

For example, if we say that the number of dimensions of theapproximation function f₂(x) is five as with the first method, with thesecond method, the features w_(i) are calculated as follows.

That is to say, FIG. 68 represents the pixel value P(x, y) shown in theleft side of Expression (45) in a graphic manner. The respective fivegraphs shown in FIG. 68 are basically the same as shown in FIG. 65.

As shown in FIG. 68, the maximal pixel values (pixel valuescorresponding to fine lines) are continuous in the direction ofcontinuity of data represented with the gradient G_(f).

Consequently, with the second method, if we supplement the respectiveinput pixel values P(x, −2), P(x, −1), P(x, 0), P(x, 1), and P(x, 2)shown in FIG. 68, for example, along the x axis, we supplement the pixelvalues after the pixel values are changed in the states shown in FIG. 69instead of supplementing the pixel values without any modification aswith the first method (let us assume that y is constant, and the fivegraphs are overlaid in the states shown in FIG. 68).

That is to say, FIG. 69 represents a state wherein the respective inputpixel values P(x, −2), P(x, −1), P(x, 0), P(x, 1), and P(x, 2) shown inFIG. 68 are shifted by the shift amount C_(x)(y) shown in the aboveExpression (43). In other words, FIG. 69 represents a state wherein thefive graphs shown in FIG. 68 are moved as if the gradient G_(F)representing the actual direction of continuity of data were regarded asa gradient G_(F)′ (in the drawing, a straight line made up of a dashedline were regarded as a straight line made up of a solid line).

In the states in FIG. 69, if we supplement the respective input pixelvalues P(x, −2), P(x, −1), P(x, 0), P(x, 1), and P(x, 2), for example,along the x axis (in the states shown in FIG. 69, if we overlay the fivegraphs), multiple lines (dashed line, broken triple-dashed line, solidline, broken line, and broken double-dashed line) in parallel with the xaxis, such as shown in FIG. 70, are distributed.

Note that in FIG. 70, the vertical axis represents pixel values, and thehorizontal axis represents a relative position x from the pixel ofinterest. Also, the dashed line represents the input pixel value P(x,−2), the broken triple-dashed line represents the input pixel value P(x,−1), the solid line represents the input pixel value P(x, 0), the brokenline represents the input pixel value P(x, 1), and the brokendouble-dashed line represents the input pixel value P(x, 2)respectively. Further, in the event of the same pixel value, lines morethan 2 lines are overlaid in reality, but in FIG. 70, the lines aredrawn so as to distinguish each line, and so as not to overlay eachline.

The respective 20 input pixel values P(x, y) (however, x is any oneinteger value of −1 through 2, and y is any one integer value of −2through 2) thus distributed, and a regression curve (the approximationfunction f₂(x) obtained by substituting the features w_(i) calculatedwith the least square method for the above Expression (38)) to minimizethe error of the value f₂(x+C_(x)(y)) become a curve f₂(x) shown in thesolid line in FIG. 70.

Thus, the approximation function f₂(x) generated with the second methodrepresents a curve connecting in the X direction the means of the inputpixel values P(x, y) in the angle θ direction (i.e., direction ofcontinuity in the general spatial direction) output from the datacontinuity detecting unit 101 (FIG. 58).

On the other hand, as described above, the approximation function f₁(x)generated with the first method represents nothing but a curveconnecting in the X direction the means of the input pixel values P(x,y) in the Y direction (i.e., the direction different from the continuityin the spatial direction).

Accordingly, as shown in FIG. 70, the approximation function f₂(x)generated with the second method becomes a function wherein the degreeof dullness of the waveform thereof decreases, and also the degree ofdecrease of the detail as to the original pixel value decreases lessthan the approximation function f₁(x) generated with the first method.In other words, though not shown in the drawing, with the approximationfunction f₂(x) generated with the second method, the waveform thereofbecomes a waveform closer to the actual X cross-sectional waveform F(x)than the approximation function f₁(x) generated with the first method.

However, as described above, the approximation function f₂(x) is afunction considering continuity in the spatial direction, but is nothingbut a function generated wherein the input image (input pixel value) isregarded as the origin (basis). That is to say, as shown in FIG. 63described above, the approximation function f₂(x) is nothing but afunction that approximated the input image different from the Xcross-sectional waveform F(x), and it is hard to say that theapproximation function f₂(x) is a function that approximated the Xcross-sectional waveform F(x). In other words, the second method is amethod for calculating the features w_(i) on assumption that the aboveExpression (44) holds, but does not take the relation in Expression (38)described above into consideration (does not consider the integrationeffects of the sensor 2).

Consequently, the present applicant has invented the third method thatcalculates the features w_(i) of the approximation function f₃(x) byfurther taking the integration effects of the sensor 2 intoconsideration as to the second method.

That is to say, the third method is a method that introduces the conceptof a spatial mixture or time mixture. Now, considering both spatialmixture and time mixture will complicate the description, so onlyspatial mixture will be considered here of spatial mixture and timemixture, and time mixture will be ignored.

Description will be made regarding spatial mixture with reference toFIG. 71 prior to description of the third method.

In FIG. 71, a portion 2321 (hereafter, referred to as a region 2321) ofa light signal in the actual world 1 represents a region having the samearea as one detecting element (pixel) of the sensor 2.

Upon the sensor 2 detecting the region 2321, the sensor 2 outputs avalue (one pixel value) 2322 obtained by the region 2321 being subjectedto integration in the temporal and spatial directions (X direction, Ydirection, and t direction). Note that the pixel value 2322 isrepresented as an image in the drawing, but is actually datarepresenting a predetermined value.

The region 2321 in the actual world 1 is clearly classified into a lightsignal (white region in the drawing) corresponding to the foreground(the above fine line, for example), and a light signal (black region inthe drawing) corresponding to the background.

On the other hand, the pixel value 2322 is a value obtained by the lightsignal in the actual world 1 corresponding to the foreground and thelight signal in the actual world 1 corresponding to the background beingsubjected to integration. In other words, the pixel value 2322 is avalue corresponding to a level wherein the light level corresponding tothe foreground and the light level corresponding to the background arespatially mixed.

Thus, in the event that a portion corresponding to one pixel (detectingelement of the sensor 2) of the light signals in the actual world 1 isnot a portion where the light signals having the same level arespatially uniformly distributed, but a portion where the light signalshaving a different level such as a foreground and background aredistributed, upon the region thereof being detected by the sensor 2, theregion becomes one pixel value as if the different light levels werespatially mixed by the integration effects of the sensor 2 (integratedin the spatial direction). Thus, a region made up of pixels in which animage (light signals in the actual world 1) corresponding to aforeground, and an image (light signals in the actual world 1)corresponding to a background are subjected to spatial integration, thatis, mixed as it were, which is spatial mixing, is referred to as aspatial mixed region here.

Accordingly, with the third method, the actual world estimating unit 102(FIG. 58) estimates the X cross-sectional waveform F(x) representing theoriginal region 2321 in the actual world 1 (of the light signals in theactual world 1, the portion 2321 corresponding to one pixel of thesensor 2) by approximating the X cross-sectional waveform F(x) with theapproximation function f₃(x) serving as a one-dimensional polynomialsuch as shown in FIG. 72.

That is to say, FIG. 72 represents an example of the approximationfunction f₃(x) corresponding to the pixel value 2322 serving as aspatial mixed region (FIG. 71), i.e., the approximation function f₃(x)that approximates the X cross-sectional waveform F(x) corresponding tothe solid line within the region 2331 in the actual world 1 (FIG. 71).In FIG. 72, the axis in the horizontal direction in the drawingrepresents an axis in parallel with the side from the upper left endx_(s) to lower right end x_(e) of the pixel corresponding to the pixelvalue 2322 (FIG. 71), which is taken as the x axis. The axis in thevertical direction in the drawing is taken as an axis representing pixelvalues.

In FIG. 72, the following Expression (46) is defined on condition thatthe result obtained by subjecting the approximation function f₃(x) tointegration in a range (pixel width) from the x_(s) to the x_(e) isgenerally identical with the pixel values P(x, y) output from the sensor2 (dependent on a margin of error e alone).

$\begin{matrix}\begin{matrix}{P = {{\int_{x_{s}}^{x_{e}}{{f_{3}(x)}{\mathbb{d}x}}} + e}} \\{= {{\int_{x_{s}}^{x_{e}}{\left( {w_{0} + {w_{1}x} + {w_{2}x^{2}} + \ldots + {w_{n}x^{n}}} \right){\mathbb{d}x}}} + e}} \\{= {{w_{0}\left( {x_{e} - x_{s}} \right)} + \ldots + {w_{n - 1}\frac{x_{e}^{n} - x_{s}^{n}}{n}} + {w_{n}\frac{x_{e}^{n + 1} - x_{s}^{n + 1}}{n + 1}} + e}}\end{matrix} & (46)\end{matrix}$

In this case, the features w_(i) of the approximation function f₃(x) arecalculated from the 20 pixel values P(x, y) (however, x is any oneinteger value of −1 through 2, and y is any one integer value of −2through 2) of the fine-line-including data region 2302 shown in FIG. 67,so the pixel value P in Expression (46) becomes the pixel values P(x,y).

Also, as with the second method, it is necessary to take continuity inthe spatial direction into consideration, and accordingly, each of thestart position x_(s) and end position x_(e) in the integral range inExpression (46) is dependent upon the shift amount C_(x)(y). That is tosay, each of the start position x_(s) and end position x_(e) of theintegral range in Expression (46) is represented such as the followingExpression (47).

$\begin{matrix}{{x_{s} = {x - {C_{x}(y)} - 0.5}}{x_{e} = {x - {C_{x}(y)} + 0.5}}} & (47)\end{matrix}$

In this case, upon each pixel value of the fine-line-including dataregion 2302 shown in FIG. 67, i.e., each of the input pixel values P(x,−2), P(x, −1), P(x, 0), P(x, 1), and P(x, 2) (however, x is any oneinteger value of −1 through 2) shown in FIG. 68 being substituted forthe above Expression (46) (the integral range is the above Expression(47)), 20 equations shown in the following Expression (48) aregenerated.

$\begin{matrix}{{{P\left( {{- 1},{- 2}} \right)} = {{\int_{{- 1} - {C_{x}{({- 2})}} - 0.5}^{{- 1} - {C_{x}{({- 2})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{1}}},{{P\left( {0,{- 2}} \right)} = {{\int_{0 - {C_{x}{({- 2})}} - 0.5}^{0 - {C_{x}{({- 2})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{2}}},{{P\left( {1,{- 2}} \right)} = {{\int_{1 - {C_{x}{({- 2})}} - 0.5}^{1 - {C_{x}{({- 2})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{3}}},{{P\left( {2,{- 2}} \right)} = {{\int_{2 - {C_{x}{({- 2})}} - 0.5}^{2 - {C_{x}{({- 2})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{4}}},{{P\left( {{- 1},{- 1}} \right)} = {{\int_{{- 1} - {C_{x}{({- 1})}} - 0.5}^{{- 1} - {C_{x}{({- 1})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{5}}},{{P\left( {0,{- 1}} \right)} = {{\int_{0 - {C_{x}{({- 1})}} - 0.5}^{0 - {C_{x}{({- 1})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{6}}},{{P\left( {1,{- 1}} \right)} = {{\int_{1 - {C_{x}{({- 1})}} - 0.5}^{1 - {C_{x}{({- 1})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{7}}},{{P\left( {2,{- 1}} \right)} = {{\int_{2 - {C_{x}{({- 1})}} - 0.5}^{2 - {C_{x}{({- 1})}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{8}}},{{P\left( {{- 1},0} \right)} = {{\int_{{- 1} - 0.5}^{{- 1} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{9}}},{{P\left( {0,0} \right)} = {{\int_{0 - 0.5}^{0 + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{10}}},{{P\left( {1,0} \right)} = {{\int_{1 - 0.5}^{1 + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{11}}},{{P\left( {2,0} \right)} = {{\int_{2 - 0.5}^{2 + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{12}}},{{P\left( {{- 1},1} \right)} = {{\int_{{- 1} - {C_{x}{(1)}} - 0.5}^{{- 1} - {C_{x}{(1)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{13}}},{{P\left( {0,1} \right)} = {{\int_{0 - {C_{x}{(1)}} - 0.5}^{0 - {C_{x}{(1)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{14}}},{{P\left( {1,1} \right)} = {{\int_{1 - {C_{x}{(1)}} - 0.5}^{1 - {C_{x}{(1)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{15}}},{{P\left( {2,1} \right)} = {{\int_{2 - {C_{x}{(1)}} - 0.5}^{2 - {C_{x}{(1)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{16}}},{{P\left( {{- 1},2} \right)} = {{\int_{{- 1} - {C_{x}{(2)}} - 0.5}^{{- 1} - {C_{x}{(2)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{17}}},{{P\left( {0,2} \right)} = {{\int_{0 - {C_{x}{(2)}} - 0.5}^{0 - {C_{x}{(2)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{18}}},{{P\left( {1,2} \right)} = {{\int_{1 - {C_{x}{(2)}} - 0.5}^{1 - {C_{x}{(2)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{19}}},{{P\left( {2,2} \right)} = {{\int_{2 - {C_{x}{(2)}} - 0.5}^{2 - {C_{x}{(2)}} + 0.5}{{f_{3}(x)}{\mathbb{d}x}}} + e_{20}}}} & (48)\end{matrix}$

Expression (48) is made up of 20 equations as with the above Expression(45). Accordingly, with the third method as with the second method, inthe event that the number of the features w_(i) of the approximationfunction f₃(x) is less than 20, i.e., in the event that theapproximation function f₃(x) is a polynomial having the number ofdimensions less than 19, for example, the features w_(i) may becalculated with the least square method. Note that the specific solutionof the least square method will be described later.

For example, if we say that the number of dimensions of theapproximation function f₃(x) is five, the approximation function f₃(x)calculated with the least square method using Expression (48) (theapproximation function f₃(x) generated with the calculated featuresw_(i)) becomes a curve shown with the solid line in FIG. 73.

Note that in FIG. 73, the vertical axis represents pixel values, and thehorizontal axis represents a relative position x from the pixel ofinterest.

As shown in FIG. 73, in the event that the approximation function f₃(x)(a curve shown with a solid line in the drawing) generated with thethird method is compared with the approximation function f₂(x) (a curveshown with a dashed line in the drawing) generated with the secondmethod, a pixel value at x=0 becomes great, and also the gradient of thecurve creates a steep waveform. This is because details increase morethan the input pixels, resulting in being unrelated to the resolution ofthe input pixels. That is to say, we can say that the approximationfunction f₃(x) approximates the X cross-sectional waveform F(x).Accordingly, though not shown in the drawing, the approximation functionf₃(x) becomes a waveform closer to the X cross-sectional waveform F(x)than the approximation function f₂(x).

FIG. 74 represents a configuration example of the actual worldestimating unit 102 employing such a one-dimensional approximatingmethod.

In FIG. 74, the actual world estimating unit 102 estimates the Xcross-sectional waveform F(x) by calculating the features w_(i) usingthe above third method (least square method) for example, and generatingthe approximation function f(x) of the above Expression (39) using thecalculated features w_(i).

As shown in FIG. 74, the actual world estimating unit 102 includes aconditions setting unit 2331, input image storage unit 2332, input pixelvalue acquiring unit 2333, integral component calculation unit 2334,normal equation generating unit 2335, and approximation functiongenerating unit 2336.

The conditions setting unit 2331 sets a pixel range (hereafter, referredto as a tap range) used for estimating the X cross-sectional waveformF(x) corresponding to a pixel of interest, and the number of dimensionsn of the approximation function f(x).

The input image storage unit 2332 temporarily stores an input image(pixel values) from the sensor 2.

The input pixel value acquiring unit 2333 acquires, of the input imagesstored in the input image storage unit 2332, an input image regioncorresponding to the tap range set by the conditions setting unit 2231,and supplies this to the normal equation generating unit 2335 as aninput pixel value table. That is to say, the input pixel value table isa table in which the respective pixel values of pixels included in theinput image region are described. Note that a specific example of theinput pixel value table will be described later.

Now, the actual world estimating unit 102 calculates the features w_(i)of the approximation function f(x) with the least square method usingthe above Expression (46) and Expression (47) here, but the aboveExpression (46) can be represented such as the following Expression(49).

$\begin{matrix}\begin{matrix}{{P\left( {x,y} \right)} = {{\sum\limits_{i = 0}^{n}{w_{i} \times \frac{\left( {x - {C_{x}(y)} + 0.5} \right)^{i + 1} - \left( {x - {C_{x}(y)} - 0.5} \right)^{i + 1}}{i + 1}}} + e}} \\{= {{\sum\limits_{i = 0}^{n}{w_{i} \times {S_{i}\left( {x_{s},x_{e}} \right)}}} + e}}\end{matrix} & (49)\end{matrix}$

In Expression (49), S_(i)(x_(s), x_(e)) represents the integralcomponents of the i-dimensional term. That is to say, the integralcomponents S_(i)(x_(s), x_(e)) are shown in the following Expression(50).

$\begin{matrix}{{S_{i}\left( {x_{s},x_{e}} \right)} = \frac{x_{e}^{i + 1} - x_{s}^{i + 1}}{i + 1}} & (50)\end{matrix}$

The integral component calculation unit 2334 calculates the integralcomponents S_(i)(x_(s), x_(e)).

Specifically, the integral components S_(i)(x_(s), x_(e)) (however, thevalue x_(s) and value x_(e) are values shown in the above Expression(46)) shown in Expression (50) may be calculated as long as the relativepixel positions (x, y), shift amount C_(x)(y), and i of thei-dimensional terms are known. Also, of these, the relative pixelpositions (x, y) are determined by the pixel of interest and the taprange, the shift amount C_(x)(y) is determined by the angle θ (by theabove Expression (41) and Expression (43)), and the range of i isdetermined by the number of dimensions n, respectively.

Accordingly, the integral component calculation unit 2334 calculates theintegral components S_(i)(x_(s), x_(e)) based on the tap range and thenumber of dimensions set by the conditions setting unit 2331, and theangle θ of the data continuity information output from the datacontinuity detecting unit 101, and supplies the calculated results tothe normal equation generating unit 2335 as an integral component table.

The normal equation generating unit 2335 generates the above Expression(46), i.e., a normal equation in the case of obtaining the featuresw_(i) of the right side of Expression (49) with the least square methodusing the input pixel value table supplied from the input pixel valueacquiring unit 2333, and the integral component table supplied from theintegral component calculation unit 2334, and supplies this to theapproximation function generating unit 2336 as a normal equation table.Note that a specific example of a normal equation will be describedlater.

The approximation function generating unit 2336 calculates therespective features w_(i) of the above Expression (49) (i.e., therespective coefficients w_(i) of the approximation function f(x) servingas a one-dimensional polynomial) by solving a normal equation includedin the normal equation table supplied from the normal equationgenerating unit 2335 using the matrix solution, and outputs these to theimage generating unit 103.

Next, description will be made regarding the actual world estimatingprocessing (processing in step S102 in FIG. 29) of the actual worldestimating unit 102 (FIG. 74) which employs the one-dimensionalapproximating method with reference to the flowchart in FIG. 75.

For example, let us say that an input image, which is a one-frame inputimage output from the sensor 2, including the fine-line-including dataregion 2302 in FIG. 60 described above has been already stored in theinput image storage unit 2332. Also, let us say that the data continuitydetecting unit 101 has subjected, at the continuity detection processingin step S101 (FIG. 29), the fine-line-including data region 2302 to theprocessing thereof, and has already output the angle θ as datacontinuity information.

In this case, the conditions setting unit 2331 sets conditions (a taprange and the number of dimensions) in step S2301 in FIG. 75.

For example, let us say that a tap range 2351 shown in FIG. 76 is set,and 5 dimensions are set as the number of dimensions.

That is to say, FIG. 76 is a diagram for describing an example of a taprange. In FIG. 76, the X direction and Y direction are the X directionand Y direction of the sensor 2 (FIG. 59) respectively. Also, the taprange 2351 represents a pixel group made up of 20 pixels in total (20squares in the drawing) of 4 pixels in the X direction, and also 5pixels in the Y direction.

Further, as shown in FIG. 76, let us say that a pixel of interest is setat the second pixel from the left and also the third pixel from thebottom in the drawing, of the tap range 2351. Also, let us say that eachpixel is denoted with a number 1 such as shown in FIG. 76 (1 is anyinteger value of 0 through 19) according to the relative pixel positions(x, y) from the pixel of interest (a coordinate value of apixel-of-interest coordinates system wherein the center (0, 0) of thepixel of interest is taken as the origin).

Now, description will return to FIG. 75, wherein in step S2302, theconditions setting unit 2331 sets a pixel of interest.

In step S2303, the input pixel value acquiring unit 2333 acquires aninput pixel value based on the condition (tap range) set by theconditions setting unit 2331, and generates an input pixel value table.That is to say, in this case, the input pixel value acquiring unit 2333acquires the fine-line-including data region 2302 (FIG. 64), andgenerates a table made up of 20 input pixel values P(l) as an inputpixel value table.

Note that in this case, the relation between the input pixel values P(l)and the above input pixel values P(x, y) is a relation shown in thefollowing Expression (51). However, in Expression (51), the left siderepresents the input pixel values P(l), and the right side representsthe input pixel values P(x, y).

$\begin{matrix}{{{P(0)} = {P\left( {0,0} \right)}}{{P(1)} = {P\left( {{- 1},2} \right)}}{{P(2)} = {P\left( {0,2} \right)}}{{P(3)} = {P\left( {1,2} \right)}}{{P(4)} = {P\left( {2,2} \right)}}{{P(5)} = {P\left( {{- 1},1} \right)}}{{P(6)} = {P\left( {0,1} \right)}}{{P(7)} = {P\left( {1,1} \right)}}{{P(8)} = {P\left( {2,1} \right)}}{{P(9)} = {P\left( {{- 1},0} \right)}}{{P(10)} = {P\left( {1,0} \right)}}{{P(11)} = {P\left( {2,0} \right)}}{{P(12)} = {P\left( {{- 1},{- 1}} \right)}}{{P(13)} = {P\left( {0,{- 1}} \right)}}{{P(14)} = {P\left( {1,{- 1}} \right)}}{{P(15)} = {P\left( {2,{- 1}} \right)}}{{P(16)} = {P\left( {{- 1},{- 2}} \right)}}{{P(17)} = {P\left( {0,{- 2}} \right)}}{{P(18)} = {P\left( {1,{- 2}} \right)}}{{P(19)} = {P\left( {2,{- 2}} \right)}}} & (51)\end{matrix}$

In step S2304, the integral component calculation unit 2334 calculatesintegral components based on the conditions (a tap range and the numberof dimensions) set by the conditions setting unit 2331, and the datacontinuity information (angle θ) supplied from the data continuitydetecting unit 101, and generates an integral component table.

In this case, as described above, the input pixel values are not P(x, y)but P(l), and are acquired as the value of a pixel number 1, so theintegral component calculation unit 2334 calculates the above integralcomponents S_(i)(x_(s), x_(e)) in Expression (50) as a function of 1such as the integral components S_(i)(l) shown in the left side of thefollowing Expression (52).

$\begin{matrix}{{S_{i}(l)} = {S_{i}\left( {x_{s},x_{e}} \right)}} & (52)\end{matrix}$

Specifically, in this case, the integral components S_(i)(l) shown inthe following Expression (53) are calculated.S _(i)(0)=S _(i)(−0.5, 0.5)  (53)S _(i)(1)=S _(i)(−1.5−C _(x)(2), −0.5−C _(x)(2))S _(i)(2)=S _(i)(−0.5−C _(x)(2), 0.5−C _(x)(2))S _(i)(3)=S _(i)(0.5−C _(x)(2), 1.5−C _(x)(2))S _(i)(4)=S _(i)(1.5−C _(x)(2), 2.5−C _(x)(2))S _(i)(5)=S _(i)(−1.5−C _(x)(1), −0.5−C _(x)(1))S _(i)(6)=S _(i)(−0.5−C _(x)(1), 0.5−C _(x)(1))S _(i)(7)=S _(i)(0.5−C _(x)(1), 1.5−C _(x)(1))S _(i)(8)=S _(i)(1.5−C _(x)(1), 2.5−C _(x)(1))S _(i)(9)=S _(i)(−1.5, −0.5)S _(i)(10)=S _(i)(0.5, 1.5)S _(i)(11)=S _(i)(1.5, 2.5)S _(i)(12)=S _(i)(−1.5−C _(x)(−1), −0.5−C _(x)(−1))S _(i)(13)=S _(i)(−0.5−C _(x)(−1), 0.5−C _(x)(−1))S _(i)(14)=S _(i)(0.5−C _(x)(−1), 1.5−C _(x)(−1))S _(i)(15)=S _(i)(1.5−C _(x)(−1), 2.5−C _(x)(−1))S _(i)(16)=S _(i)(−1.5−C _(x)(−2), −0.5−C _(x)(−2))S _(i)(17)=S _(i)(−0.5−C _(x)(−2), 0.5−C _(x)(−2))S _(i)(18)=S _(i)(0.5−C _(x)(−2), 1.5−C _(x)(−2))S _(i)(19)=S _(i)(1.5−C _(x)(−2), 2.5−C _(x)(−2))

Note that in Expression (53), the left side represents the integralcomponents S_(i)(l), and the right side represents the integralcomponents S_(i)(x_(s), x_(e)). That is to say, in this case, i is 0through 5, and accordingly, the 120 S_(i)(l) in total of the 20 S₀(l),20 S_(i)(l), 20 S₂(l) 20 S₃(l), 20 S₄(l), and 20 S₅(l) are calculated.

More specifically, first the integral component calculation unit 2334calculates each of the shift amounts C_(x)(−2), C_(x)(−1), C_(x)(1), andC_(x)(2) using the angle θ supplied from the data continuity detectingunit 101. Next, the integral component calculation unit 2334 calculateseach of the 20 integral components S_(i)(x_(s), x_(e)) shown in theright side of Expression (52) regarding each of i=0 through 5 using thecalculated shift amounts C_(x)(−2), C_(x)(−1), C_(x)(1), and C_(x)(2).That is to say, the 120 integral components S_(i) (x_(s), x_(e)) arecalculated. Note that with this calculation of the integral componentsS_(i)(x_(s), x_(e)), the above Expression (50) is used. Subsequently,the integral component calculation unit 2334 converts each of thecalculated 120 integral components S_(i)(x_(s), x_(e)) into thecorresponding integral components S_(i)(l) in accordance with Expression(53), and generates an integral component table including the converted120 integral components S_(i)(l).

Note that the sequence of the processing in step S2303 and theprocessing in step S2304 is not restricted to the example in FIG. 75,the processing in step S2304 may be executed first, or the processing instep S2303 and the processing in step S2304 may be executedsimultaneously.

Next, in step S2305, the normal equation generating unit 2335 generatesa normal equation table based on the input pixel value table generatedby the input pixel value acquiring unit 2333 at the processing in stepS2303, and the integral component table generated by the integralcomponent calculation unit 2334 at the processing in step S2304.

Specifically, in this case, the features w_(i) of the followingExpression (54) corresponding to the above Expression (49) arecalculated using the least square method. A normal equationcorresponding to this is represented as the following Expression (55).

$\begin{matrix}{{P(I)} = {{\sum\limits_{i = 0}^{n}{w_{i} \times {S_{i}(I)}}} + e}} & (54) \\{{\begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}\begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (55)\end{matrix}$

Note that in Expression (55), L represents the maximum value of thepixel number 1 in the tap range. n represents the number of dimensionsof the approximation function f(x) serving as a polynomial.Specifically, in this case, n=5, and L=19.

If we define each matrix of the normal equation shown in Expression (55)as the following Expressions (56) through (58), the normal equation isrepresented as the following Expression (59).

$\begin{matrix}{S_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}} & (56) \\{W_{MAT} = \begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} & (57) \\{P_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (58) \\{{S_{MAT}W_{MAT}} = P_{MAT}} & (59)\end{matrix}$

As shown in Expression (57), the respective components of the matrixW_(MAT) are the features w_(i) to be obtained. Accordingly, inExpression (59), if the matrix S_(MAT) of the left side and the matrixP_(MAT) of the right side are determined, the matrix W_(MAT) (i.e.,features w_(i)) may be calculated with the matrix solution.

Specifically, as shown in Expression (56), the respective components ofthe matrix S_(MAT) may be calculated as long as the above integralcomponents S_(i)(l) are known. The integral components S_(i)(l) areincluded in the integral component table supplied from the integralcomponent calculation unit 2334, so the normal equation generating unit2335 can calculate each component of the matrix S_(MAT) using theintegral component table.

Also, as shown in Expression (58), the respective components of thematrix P_(MAT) may be calculated as long as the integral componentsS_(i)(l) and the input pixel values P(l) are known. The integralcomponents S_(i)(l) is the same as those included in the respectivecomponents of the matrix S_(MAT), also the input pixel values P(l) areincluded in the input pixel value table supplied from the input pixelvalue acquiring unit 2333, so the normal equation generating unit 2335can calculate each component of the matrix P_(MAT) using the integralcomponent table and input pixel value table.

Thus, the normal equation generating unit 2335 calculates each componentof the matrix S_(MAT) and matrix P_(MAT), and outputs the calculatedresults (each component of the matrix S_(MAT) and matrix P_(MAT)) to theapproximation function generating unit 2336 as a normal equation table.

Upon the normal equation table being output from the normal equationgenerating unit 2335, in step S2306, the approximation functiongenerating unit 2336 calculates the features w_(i)(i.e., thecoefficients w_(i) of the approximation function f(x) serving as aone-dimensional polynomial) serving as the respective components of thematrix W_(MAT) in the above Expression (59) based on the normal equationtable.

Specifically, the normal equation in the above Expression (59) can betransformed as the following Expression (60).

$\begin{matrix}{\left| W_{MAT} \right. = {S_{MAT}^{- 1}P_{MAT}}} & (60)\end{matrix}$

In Expression (60), the respective components of the matrix W_(MAT) inthe left side are the features w_(i) to be obtained. The respectivecomponents regarding the matrix S_(MAT) and matrix P_(MAT) are includedin the normal equation table supplied from the normal equationgenerating unit 2335. Accordingly, the approximation function generatingunit 2336 calculates the matrix W_(MAT) by calculating the matrix in theright side of Expression (60) using the normal equation table, andoutputs the calculated results (features w_(i)) to the image generatingunit 103.

In step S2307, the approximation function generating unit 2336determines regarding whether or not the processing of all the pixels hasbeen completed.

In step S2307, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S2302, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S2302 through S2307 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S2307, in the event that determination is made that theprocessing of all the pixels has been completed), the estimatingprocessing of the actual world 1 ends.

Note that the waveform of the approximation function f(x) generated withthe coefficients (features) w_(i) thus calculated becomes a waveformsuch as the approximation function f₃(x) in FIG. 73 described above.

Thus, with the one-dimensional approximating method, the features of theapproximation function f(x) serving as a one-dimensional polynomial arecalculated under the assumption, for example, that a waveform having thesame form as the one-dimensional X cross-sectional waveform F(x) iscontinuous in the direction of continuity. Accordingly, with theone-dimensional approximating method, the features of the approximationfunction f(x) can be calculated with less amount of calculationprocessing than other function approximating methods.

Next, description will be made regarding the second functionapproximating method with reference to FIG. 77 through FIG. 83.

That is to say, the second function approximating method is a methodwherein the light signal in the actual world 1 having continuity in thespatial direction represented with the gradient G_(F) such as shown inFIG. 77 for example is regarded as a waveform F(x, y) on the X-Y plane(on the plane level in the X direction serving as one direction of thespatial directions, and in the Y direction orthogonal to the Xdirection), and the waveform F(x, y) is approximated with theapproximation function f(x, y) serving as a two-dimensional polynomial,thereby estimating the waveform F(x, y). Accordingly, hereafter, thesecond function approximating method is referred to as a two-dimensionalpolynomial approximating method.

Note that in FIG. 77, the horizontal direction represents the Xdirection serving as one direction of the spatial directions, the upperright direction represents the Y direction serving as the otherdirection of the spatial directions, and the vertical directionrepresents the level of light respectively. G_(F) represents thegradient as continuity in the spatial direction.

Also, with description of the two-dimensional polynomial approximatingmethod, let us say that the sensor 2 is a CCD made up of the multipledetecting elements 2-1 disposed on the plane thereof, such as shown inFIG. 78.

With the example in FIG. 78, the direction in parallel with apredetermined side of the detecting elements 2-1 is taken as the Xdirection serving as one direction of the spatial directions, and thedirection orthogonal to the X direction is taken as the Y directionserving as the other direction of the spatial directions. The directionorthogonal to the X-Y plane is taken as the t direction serving as thetemporal direction.

Also, with the example in FIG. 78, the spatial shape of the respectivedetecting elements 2-1 of the sensor 2 is taken as a square of which oneside is 1 in length. The shutter time (exposure time) of the sensor 2 istaken as 1.

Further, with the example in FIG. 78, the center of one certaindetecting element 2-1 of the sensor 2 is taken as the origin (theposition in the X direction is x=0, and the position in the Y directionis y=0) in the spatial directions (X direction and Y direction), andalso the intermediate point-in-time of the exposure time is taken as theorigin (the position in the t direction is t=0) in the temporaldirection (t direction).

In this case, the detecting element 2-1 of which the center is in theorigin (x=0, y=0) in the spatial directions subjects the light signalfunction F(x, y, t) to integration with a range of −0.5 through 0.5 inthe X direction, with a range of −0.5 through 0.5 in the Y direction,and with a range of −0.5 through 0.5 in the t direction, and outputs theintegral value as the pixel value P.

That is to say, the pixel value P output from the detecting element 2-1of which the center is in the origin in the spatial directions isrepresented with the following Expression (61).

$\begin{matrix}{P = {\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{{F\left( {x,y,t} \right)}\ {\mathbb{d}x}\ {\mathbb{d}y}\ {\mathbb{d}t}}}}}} & (61)\end{matrix}$

Similarly, the other detecting elements 2-1 output the pixel value Pshown in Expression (61) by taking the center of the detecting element2-1 to be processed as the origin in the spatial directions.

Incidentally, as described above, the two-dimensional polynomialapproximating method is a method wherein the light signal in the actualworld 1 is handled as a waveform F(x, y) such as shown in FIG. 77 forexample, and the two-dimensional waveform F(x, y) is approximated withthe approximation function f(x, y) serving as a two-dimensionalpolynomial.

First, description will be made regarding a method representing such theapproximation function f(x, y) with a two-dimensional polynomial.

As described above, the light signal in the actual world 1 isrepresented with the light signal function F(x, y, t) of which variablesare the position on the three-dimensional space x, y, and z, andpoint-in-time t. This light signal function F(x, y, t), i.e., aone-dimensional waveform projected in the X direction at an arbitraryposition y in the Y direction is referred to as an X cross-sectionalwaveform F(x), here.

When paying attention to this X cross-sectional waveform F(x), in theevent that the signal in the actual world 1 has continuity in a certaindirection in the spatial directions, it can be conceived that a waveformhaving the same form as the X cross-sectional waveform F(x) continues inthe continuity direction. For example, with the example in FIG. 77, awaveform having the same form as the X cross-sectional waveform F(x)continues in the direction of the gradient G_(F). In other words, it canbe said that the waveform F(x, y) is formed by a waveform having thesame form as the X cross-sectional waveform F(x) continuing in thedirection of the gradient G_(F).

Accordingly, the approximation function f(x, y) can be represented witha two-dimensional polynomial by considering that the waveform of theapproximation function f(x, y) approximating the waveform F(x, y) isformed by a waveform having the same form as the approximation functionf(x) approximating the X cross-sectional F(x) continuing.

Description will be made in more detail regarding the representingmethod of the approximation function f(x, y).

For example, let us say that the light signal in the actual world 1 suchas shown in FIG. 77 described above, i.e., a light signal havingcontinuity in the spatial direction represented with the gradient G_(F)is detected by the sensor 2 (FIG. 78), and output as an input image(pixel value).

Further, let us say that as shown in FIG. 79, the data continuitydetecting unit 101 (FIG. 3) subjects an input image region 2401 made upof 20 pixels (in the drawing, 20 squares represented with dashed line)in total of 4 pixels in the X direction and also 5 pixels in the Ydirection, of this input image, to the processing thereof, and outputsan angle θ (angle θ generated between the direction of data continuityrepresented with the gradient G_(f) corresponding to the gradient G_(F),and the X direction) as one of the data continuity information.

Note that with the input image region 2401, the horizontal direction inthe drawing represents the X direction serving as one direction in thespatial directions, and the vertical direction in the drawing representsthe Y direction serving as the other direction of the spatialdirections.

Also, in FIG. 79, an (x, y) coordinates system is set such that a pixelin the second pixel from the left, and also the third pixel from thebottom is taken as a pixel of interest, and the center of the pixel ofinterest is taken as the origin (0, 0). A relative distance (hereafter,referred to as a cross-sectional direction distance) in the X directionas to the straight line (straight line having the gradient G_(f)representing the direction of data continuity) having an angle θ passingthrough the origin (0, 0) is described as x′.

Further, in FIG. 79, the graph on the right side is a function whereinan X cross-sectional waveform F(x′) is approximated, which represents anapproximation function f(x′) serving as an n-dimensional (n is anarbitrary integer) polynomial. Of the axes in the graph on the rightside, the axis in the horizontal direction in the drawing represents across-sectional direction distance, and the axis in the verticaldirection in the drawing represents pixel values.

In this case, the approximation function f(x′) shown in FIG. 79 is ann-dimensional polynomial, so is represented as the following Expression(62).

$\begin{matrix}{{f\left( x^{\prime} \right)} = {{w_{0} + {w_{1}x^{\prime}} + {w_{2}x^{\prime}} + \ldots + {w_{n}x^{\prime\; n}}} = {\sum\limits_{i = 0}^{n}{w_{i}x^{\prime\; i}}}}} & (62)\end{matrix}$

Also, since the angle θ is determined, the straight line having angle θpassing through the origin (0, 0) is uniquely determined, and a positionx₁ in the X direction of the straight line at an arbitrary position y inthe Y direction is represented as the following Expression (63).However, in Expression (63), s represents cot θ (=1{tilde over ( )}□□□θ).x _(l) =s×y  (63)

That is to say, as shown in FIG. 79, a point on the straight linecorresponding to continuity of data represented with the gradient G_(f)is represented with a coordinate value (x₁, y).

The cross-sectional direction distance x′ is represented as thefollowing Expression (64) using Expression (63).x′=x−x _(l) =x−s×y  (64)

Accordingly, the approximation function f(x, y) at an arbitrary position(x, y) within the input image region 2401 is represented as thefollowing Expression (65) using Expression (62) and Expression (64).

$\begin{matrix}{{f\left( {x,y} \right)} = {\sum\limits_{i = 0}^{n}{w_{i}\left( {x - {s \times y}} \right)}^{i}}} & (65)\end{matrix}$

Note that in Expression (65), w_(i) represents coefficients of theapproximation function f(x, y). Note that the coefficients w_(i) of theapproximation function f including the approximation function f(x, y)can be evaluated as the features of the approximation function f.Accordingly, the coefficients w_(i) of the approximation function f arealso referred to as the features w_(i) of the approximation function f.

Thus, the approximation function f(x, y) having a two-dimensionalwaveform can be represented as the polynomial of Expression (65) as longas the angle θ is known.

Accordingly, if the actual world estimating unit 102 can calculate thefeatures w_(i) of Expression (65), the actual world estimating unit 102can estimate the waveform F(x, y) such as shown in FIG. 77.

Consequently, hereafter, description will be made regarding a method forcalculating the features w_(i) of Expression (65).

That is to say, upon the approximation function f(x, y) represented withExpression (65) being subjected to integration with an integral range(integral range in the spatial direction) corresponding to a pixel (thedetecting element 2-1 of the sensor 2 (FIG. 78)), the integral valuebecomes the estimated value regarding the pixel value of the pixel. Itis the following Expression (66) that this is represented with anequation. Note that with the two-dimensional polynomial approximatingmethod, the temporal direction t is regarded as a constant value, soExpression (66) is taken as an equation of which variables are thepositions x and y in the spatial directions (X direction and Ydirection).

$\begin{matrix}{{{P\left( {x,y} \right)} = {{\int_{y - 0.5}^{y + 0.5}{\int_{x - 0.5}^{x + 0.5}{\sum\limits_{i = 0}^{n}{w_{i}\left( {x - {s \times y}} \right)}^{i}}}} + e}}\ } & (66)\end{matrix}$

In Expression (66), P(x, y) represents the pixel value of a pixel ofwhich the center position is in a position (x, y) (relative position (x,y) from the pixel of interest) of an input image from the sensor 2.Also, e represents a margin of error.

Thus, with the two-dimensional polynomial approximating method, therelation between the input pixel value P(x, y) and the approximationfunction f(x, y) serving as a two-dimensional polynomial can berepresented with Expression (66), and accordingly, the actual worldestimating unit 102 can estimate the two-dimensional function F(x, y)(waveform F(x, y) wherein the light signal in the actual world 1 havingcontinuity in the spatial direction represented with the gradient G_(F)(FIG. 77) is represented focusing attention on the spatial direction) bycalculating the features w_(i) with, for example, the least squaremethod or the like using Expression (66) (by generating theapproximation function f(x, y) by substituting the calculated featuresw_(i) for Expression (64)).

FIG. 80 represents a configuration example of the actual worldestimating unit 102 employing such a two-dimensional polynomialapproximating method.

As shown in FIG. 80, the actual world estimating unit 102 includes aconditions setting unit 2421, input image storage unit 2422, input pixelvalue acquiring unit 2423, integral component calculation unit 2424,normal equation generating unit 2425, and approximation functiongenerating unit 2426.

The conditions setting unit 2421 sets a pixel range (tap range) used forestimating the function F(x, y) corresponding to a pixel of interest,and the number of dimensions n of the approximation function f(x, y).

The input image storage unit 2422 temporarily stores an input image(pixel values) from the sensor 2.

The input pixel value acquiring unit 2423 acquires, of the input imagesstored in the input image storage unit 2422, an input image regioncorresponding to the tap range set by the conditions setting unit 2421,and supplies this to the normal equation generating unit 2425 as aninput pixel value table. That is to say, the input pixel value table isa table in which the respective pixel values of pixels included in theinput image region are described. Note that a specific example of theinput pixel value table will be described later.

Incidentally, as described above, the actual world estimating unit 102employing the two-dimensional approximating method calculates thefeatures w_(i) of the approximation function f(x, y) represented withthe above Expression (65) by solving the above Expression (66) using theleast square method.

Expression (66) can be represented as the following Expression (71) byusing the following Expression (70) obtained by the followingExpressions (67) through (69).

$\begin{matrix}{{\int{x^{i}{\mathbb{d}x}}} = \frac{x^{i + 1}}{i + 1}} & (67) \\{{\int{\left( {x - {s \times y}} \right)^{i}{\mathbb{d}x}}} = \frac{\left( {x - {s \times y}} \right)^{i + 1}}{\left( {i + 1} \right)}} & (68) \\{{\int{\left( {x - {s \times y}} \right)^{i}{\mathbb{d}y}}} = \frac{\left( {x - {s \times y}} \right)^{i + 1}}{s\left( {i + 1} \right)}} & (69) \\{{\int_{y - 0.5}^{y + 0.5}{\int_{x - 0.5}^{x + 0.5}{\left( {x - {s \times y}} \right)^{i}\ {\mathbb{d}x}\ {\mathbb{d}y}}}} = {{\int_{y - 0.5}^{y + 0.5}{\left\lbrack \frac{\left( {x - {s \times y}} \right)^{i + 1}}{\left( {i + 1} \right)} \right\rbrack_{x - 0.5}^{x + 0.5}\ {\mathbb{d}y}}} = {{\int_{y - 0.5}^{y + 0.5}{\frac{\left( {x + 0.5 - {s \times y}} \right)^{i + 1} - \left( {x - 0.5 - {s \times y}} \right)^{i + 1}}{i + 1}\ {\mathbb{d}y}}} = {{\left\lbrack \frac{\left( {x + 0.5 - {s \times y}} \right)^{i + 2}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)} \right\rbrack_{y - 0.5}^{y + 0.5} - \left\lbrack \frac{\left( {x - 0.5 - {s \times y}} \right)^{i + 2}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)} \right\rbrack_{y - 0.5}^{y + 0.5}} = \frac{\begin{matrix}{\left( {x + 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} - \left( {x + 0.5 - {s \times y} - {0.5s}} \right)^{i + 2} -} \\{\left( {x - 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} + \left( {x - 0.5 - {s \times y} - {0.5s}} \right)^{i + 2}}\end{matrix}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)}}}}} & (70) \\\begin{matrix}{{P\left( {x,y} \right)} = {\sum\limits_{i = 0}^{n}{\frac{w_{i}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)}\left\{ {\left( {x + 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} -} \right.}}} \\{\left( {x + 0.5 - {s \times y} - {0.5s}} \right)^{i + 2} - \left( {x - 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} +} \\{\left. \left( {x - 0.5 - {s \times y} - {0.5s}} \right)^{i + 2} \right\} + e} \\{= {{\sum\limits_{i = 0}^{n}{w_{i}{s_{i}\left( {{x - 0.5},{x + 0.5},{y - 0.5},{y + 0.5}} \right)}}} + e}}\end{matrix} & (71)\end{matrix}$

In Expression (71), S_(i)(x−0.5, x+0.5, y−0.5, y+0.5) represents theintegral components of i-dimensional terms. That is to say, the integralcomponents S_(i)(x−0.5, x+0.5, y−0.5, y+0.5) are as shown in thefollowing Expression (72).

$\begin{matrix}{{s_{i}\begin{pmatrix}{{x - 0.5},{x + 0.5},} \\{{y - 0.5},{y + 0.5}}\end{pmatrix}} = \frac{\begin{matrix}{\left( {x + 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} - \left( {x + 0.5 - {s \times y} - {0.5s}} \right)^{i + 2} -} \\{\left( {x - 0.5 - {s \times y} + {0.5s}} \right)^{i + 2} + \left( {x - 0.5 - {s \times y} - {0.5s}} \right)^{i + 2}}\end{matrix}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)}} & (72)\end{matrix}$

The integral component calculation unit 2424 calculates the integralcomponents S_(i)(x−0.5, x+0.5, y−0.5, y+0.5).

Specifically, the integral components S_(i)(x−0.5, x+0.5, y−0.5, y+0.5)shown in Expression (72) can be calculated as long as the relative pixelpositions (x, y), the variable s and i of i-dimensional terms in theabove Expression (65) are known. Of these, the relative pixel positions(x, y) are determined with a pixel of interest, and a tap range, thevariable s is cot θ, which is determined with the angle θ, and the rangeof i is determined with the number of dimensions n respectively.

Accordingly, the integral component calculation unit 2424 calculates theintegral components S_(i)(x−0.5, x+0.5, y−0.5, y+0.5) based on the taprange and the number of dimensions set by the conditions setting unit2421, and the angle θ of the data continuity information output from thedata continuity detecting unit 101, and supplies the calculated resultsto the normal equation generating unit 2425 as an integral componenttable.

The normal equation generating unit 2425 generates a normal equation inthe case of obtaining the above Expression (66), i.e., Expression (71)by the least square method using the input pixel value table suppliedfrom the input pixel value acquiring unit 2423, and the integralcomponent table supplied from the integral component calculation unit2424, and outputs this to the approximation function generating unit2426 as a normal equation table. Note that a specific example of anormal equation will be described later.

The approximation function generating unit 2426 calculates therespective features w_(i) of the above Expression (66) (i.e., thecoefficients w_(i) of the approximation function f(x, y) serving as atwo-dimensional polynomial) by solving the normal equation included inthe normal equation table supplied from the normal equation generatingunit 2425 using the matrix solution, and output these to the imagegenerating unit 103.

Next, description will be made regarding the actual world estimatingprocessing (processing in step S102 in FIG. 29) to which thetwo-dimensional polynomial approximating method is applied, withreference to the flowchart in FIG. 81.

For example, let us say that the light signal in the actual world 1having continuity in the spatial direction represented with the gradientG_(F) has been detected by the sensor 2 (FIG. 78), and has been storedin the input image storage unit 2422 as an input image corresponding toone frame. Also, let us say that the data continuity detecting unit 101has subjected the region 2401 shown in FIG. 79 described above of theinput image to processing in the continuity detecting processing in stepS101 (FIG. 29), and has output the angle θ as data continuityinformation.

In this case, in step S2401, the conditions setting unit 2421 setsconditions (a tap range and the number of dimensions).

For example, let us say that a tap range 2441 shown in FIG. 82 has beenset, and also 5 has been set as the number of dimensions.

FIG. 82 is a diagram for describing an example of a tap range. In FIG.82, the X direction and Y direction represent the X direction and Ydirection of the sensor 2 (FIG. 78). Also, the tap range 2441 representsa pixel group made up of 20 pixels (20 squares in the drawing) in totalof 4 pixels in the X direction and also 5 pixels in the Y direction.

Further, as shown in FIG. 82, let us say that a pixel of interest hasbeen set to a pixel, which is the second pixel from the left and alsothe third pixel from the bottom in the drawing, of the tap range 2441.Also, let us say that each pixel is denoted with a number 1 such asshown in FIG. 82 (l is any integer value of 0 through 19) according tothe relative pixel positions (x, y) from the pixel of interest (acoordinate value of a pixel-of-interest coordinates system wherein thecenter (0, 0) of the pixel of interest is taken as the origin).

Now, description will return to FIG. 81, wherein in step S2402, theconditions setting unit 2421 sets a pixel of interest.

In step S2403, the input pixel value acquiring unit 2423 acquires aninput pixel value based on the condition (tap range) set by theconditions setting unit 2421, and generates an input pixel value table.That is to say, in this case, the input pixel value acquiring unit 2423acquires the input image region 2401 (FIG. 79), generates a table madeup of 20 input pixel values P(l) as an input pixel value table.

Note that in this case, the relation between the input pixel values P(l)and the above input pixel values P(x, y) is a relation shown in thefollowing Expression (73). However, in Expression (73), the left siderepresents the input pixel values P(l), and the right side representsthe input pixel values P(x, y).

$\begin{matrix}{{{{P(0)} = {P\left( {0,0} \right)}}{{P(1)} = {P\left( {{- 1},2} \right)}}{{P(2)} = {P\left( {0,2} \right)}}{{P(3)} = {P\left( {1,2} \right)}}{{P(4)} = {P\left( {2,2} \right)}}{{P(5)} = {P\left( {{- 1},1} \right)}}{{P(6)} = {P\left( {0,1} \right)}}{{P(7)} = {P\left( {1,1} \right)}}{{P(8)} = {P\left( {2,1} \right)}}{{P(9)} = {P\left( {{- 1},0} \right)}}{{P(10)} = {P\left( {1,0} \right)}}{{P(11)} = {P\left( {2,0} \right)}}{{P(12)} = {P\left( {{- 1},{- 1}} \right)}}{{P(13)} = {P\left( {0,{- 1}} \right)}}{{P(14)} = {P\left( {1,{- 1}} \right)}}{P(15)} = {P\left( {2,{- 1}} \right)}}{{P(16)} = {P\left( {{- 1},{- 2}} \right)}}{{P(17)} = {P\left( {0,{- 2}} \right)}}{{P(18)} = {P\left( {1,{- 2}} \right)}}{{P(19)} = {P\left( {2,{- 2}} \right)}}} & (73)\end{matrix}$

In step S2404, the integral component calculation unit 2424 calculatesintegral components based on the conditions (a tap range and the numberof dimensions) set by the conditions setting unit 2421, and the datacontinuity information (angle θ) supplied from the data continuitydetecting unit 101, and generates an integral component table.

In this case, as described above, the input pixel values are not P(x, y)but P(l), and are acquired as the value of a pixel number l, so theintegral component calculation unit 2424 calculates the integralcomponents Si (x−0.5, x+0.5, y−0.5, y+0.5) in the above Expression (72)as a function of 1 such as the integral components S_(i)(l) shown in theleft side of the following Expression (74).

$\begin{matrix}{{S_{i}(l)} = {S_{i}\left( {{x - 0.5},{x + 0.5},{y - 0.5},{y + 0.5}} \right)}} & (74)\end{matrix}$

Specifically, in this case, the integral components S_(i)(l) shown inthe following Expression (75) are calculated.

$\begin{matrix}{{{{S_{i}(0)} = {S_{i}\left( {{- 0.5},0.5,{- 0.5},0.5} \right)}}{S_{i}(1)} = {S_{i}\left( {{- 1.5},{- 0.5},1.5,2.5} \right)}}{{S_{i}(2)} = {S_{i}\left( {{- 0.5},0.5,1.5,2.5} \right)}}{{S_{i}(3)} = {S_{i}\left( {0.5,1.5,1.5,2.5} \right)}}{{S_{i}(4)} = {S_{i}\left( {1.5,2.5,1.5,2.5} \right)}}{{S_{i}(5)} = {S_{i}\left( {{- 1.5},{- 0.5},0.5,1.5} \right)}}{{S_{i}(6)} = {S_{i}\left( {{- 0.5},0.5,0.5,1.5} \right)}}{{S_{i}(7)} = {S_{i}\left( {0.5,1.5,0.5,1.5} \right)}}{{S_{i}(8)} = {S_{i}\left( {1.5,2.5,0.5,1.5} \right)}}{{S_{i}(9)} = {S_{i}\left( {{- 1.5},{- 0.5},{- 0.5},0.5} \right)}}{{S_{i}(10)} = {S_{i}\left( {0.5,1.5,{- 0.5},0.5} \right)}}{{S_{i}(11)} = {S_{i}\left( {1.5,2.5,{- 0.5},0.5} \right)}}{{S_{i}(12)} = {S_{i}\left( {{- 1.5},{- 0.5},{- 1.5},{- 0.5}} \right)}}{{S_{i}(13)} = {S_{i}\left( {{- 0.5},0.5,{- 1.5},{- 0.5}} \right)}}{{S_{i}(14)} = {S_{i}\left( {0.5,1.5,{- 1.5},{- 0.5}} \right)}}{{S_{i}(15)} = {S_{i}\left( {1.5,2.5,{- 1.5},{- 0.5}} \right)}}{{S_{i}(16)} = {S_{i}\left( {{- 1.5},{- 0.5},{- 2.5},1.5} \right)}}{{S_{i}(17)} = {S_{i}\left( {{- 0.5},0.5,{- 2.5},{- 1.5}} \right)}}{{S_{i}(18)} = {S_{i}\left( {0.5,1.5,{- 2.5},1.5} \right)}}{{S_{i}(19)} = {S_{i}\left( {1.5,2.5,{- 2.5},{- 1.5}} \right)}}} & (75)\end{matrix}$

Note that in Expression (75), the left side represents the integralcomponents S_(i)(l), and the right side represents the integralcomponents S_(i)(x−0.5, x+0.5, y−0.5, y+0.5). That is to say, in thiscase, i is 0 through 5, and accordingly, the 120 S_(i)(l) in total ofthe 20 S₀(l), 20 S₁(l), 20 S₂(l), 20 S₃(l), 20 S₄(l), and 20 S₅(l) arecalculated.

More specifically, first the integral component calculation unit 2424calculates cot θ corresponding to the angle θ supplied from the datacontinuity detecting unit 101, and takes the calculated result as avariable s. Next, the integral component calculation unit 2424calculates each of the 20 integral components S_(i)(x−0.5, x+0.5, y−0.5,y+0.5) shown in the right side of Expression (74) regarding each of i=0through 5 using the calculated variable s. That is to say, the 120integral components S_(i)(x−0.5, x+0.5, y−0.5, y+0.5) are calculated.For computing the integration component S_(i)(x−0.5, x+0.5, y−0.5,y+0.5), the above-described Expression (72) is used. Also theintegration component computing unit 2424 converts each of the 120computed integration components S_(i)(x−0.5, x+0.5, y−0.5, y+0.5) intothe corresponding S_(i)(l) respectively, according to Expression (75),and generates an integration component table containing the 120converted S_(i)(l).

Note that the sequence of the processing in step S2403 and theprocessing in step S2404 is not restricted to the example in FIG. 81,the processing in step S2404 may be executed first, or the processing instep S2403 and the processing in step S2404 may be executedsimultaneously.

Next, in step S2405, the normal equation generating unit 2425 generatesa normal equation table based on the input pixel value table generatedby the input pixel value acquiring unit 2423 at the processing in stepS2403, and the integral component table generated by the integralcomponent calculation unit 2424 at the processing in step S2404.

Specifically, in this case, the features w_(i) are calculated with theleast square method using the above Expression (71) (however, inExpression (70), the S_(i)(l) into which the integral componentsS_(i)(x−0.5, x+0.5, y−0.5, y+0.5) are converted using Expression (74) isused), so a normal equation corresponding to this is represented as thefollowing Expression (76).

$\begin{matrix}{{\begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}\begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (76)\end{matrix}$

Note that in Expression (76), L represents the maximum value of thepixel number l in the tap range. n represents the number of dimensionsof the approximation function f(x) serving as a polynomial.Specifically, in this case, n=5, and L=19.

If we define each matrix of the normal equation shown in Expression (76)as the following Expressions (77) through (79), the normal equation isrepresented as the following Expression (80).

$\begin{matrix}{S_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}} & (77) \\{W_{MAT} = \begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} & (78) \\{P_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (79) \\{{S_{MAT}W_{MAT}} = P_{MAT}} & (80)\end{matrix}$

As shown in Expression (78), the respective components of the matrixW_(MAT) are the features w_(i) to be obtained. Accordingly, inExpression (80), if the matrix S_(MAT) of the left side and the matrixP_(MAT) of the right side are determined, the matrix W_(MAT) may becalculated with the matrix solution.

Specifically, as shown in Expression (77), the respective components ofthe matrix S_(MAT) may be calculated with the above integral componentsS_(i)(l). That is to say, the integral components S_(i)(l) are includedin the integral component table supplied from the integral componentcalculation unit 2424, so the normal equation generating unit 2425 cancalculate each component of the matrix S_(MAT) using the integralcomponent table.

Also, as shown in Expression (79), the respective components of thematrix P_(MAT) may be calculated with the integral components S_(i)(l)and the input pixel values P(l) That is to say, the integral componentsS_(i)(l) is the same as those included in the respective components ofthe matrix S_(MAT), also the input pixel values P(l) are included in theinput pixel value table supplied from the input pixel value acquiringunit 2423, so the normal equation generating unit 2425 can calculateeach component of the matrix P_(MAT) using the integral component tableand input pixel value table.

Thus, the normal equation generating unit 2425 calculates each componentof the matrix S_(MAT) and matrix P_(MAT), and outputs the calculatedresults (each component of the matrix S_(MAT) and matrix P_(MAT)) to theapproximation function generating unit 2426 as a normal equation table.

Upon the normal equation table being output from the normal equationgenerating unit 2425, in step S2406, the approximation functiongenerating unit 2426 calculates the features w_(i) (i.e., thecoefficients w_(i) of the approximation function f(x, y) serving as atwo-dimensional polynomial) serving as the respective components of thematrix W_(MAT) in the above Expression (80) based on the normal equationtable.

Specifically, the normal equation in the above Expression (80) can betransformed as the following Expression (81).

$\begin{matrix}{W_{MAT} = {S_{MAT}^{- 1}P_{MAT}}} & (81)\end{matrix}$

In Expression (81), the respective components of the matrix W_(MAT) inthe left side are the features w_(i) to be obtained. The respectivecomponents regarding the matrix S_(MAT) and matrix P_(MAT) are includedin the normal equation table supplied from the normal equationgenerating unit 2425. Accordingly, the approximation function generatingunit 2426 calculates the matrix W_(MAT) by calculating the matrix in theright side of Expression (81) using the normal equation table, andoutputs the calculated results (features w_(i)) to the image generatingunit 103.

In step S2407, the approximation function generating unit 2426determines regarding whether or not the processing of all the pixels hasbeen completed.

In step S2407, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S2402, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S2402 through S2407 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S2407, in the event that determination is made that theprocessing of all the pixels has been completed), the estimatingprocessing of the actual world 1 ends.

As description of the two-dimensional polynomial approximating method,an example for calculating the coefficients (features) w_(i) of theapproximation function f(x, y) corresponding to the spatial directions(X direction and Y direction) has been employed, but the two-dimensionalpolynomial approximating method can be applied to the temporal andspatial directions (X direction and t direction, or Y direction and tdirection) as well.

That is to say, the above example is an example in the case of the lightsignal in the actual world 1 having continuity in the spatial directionrepresented with the gradient G_(F) (FIG. 77), and accordingly, theequation including two-dimensional integration in the spatial directions(X direction and Y direction), such as shown in the above Expression(66). However, the concept regarding two-dimensional integration can beapplied not only to the spatial direction but also to the temporal andspatial directions (X direction and t direction, or Y direction and tdirection).

In other words, with the two-dimensional polynomial approximatingmethod, even in the case in which the light signal function F(x, y, t),which needs to be estimated, has not only continuity in the spatialdirection but also continuity in the temporal and spatial directions(however, X direction and t direction, or Y direction and t direction),this can be approximated with a two-dimensional approximation functionf.

Specifically, for example, in the event that there is an object movinghorizontally in the X direction at uniform velocity, the direction ofmovement of the object is represented with like a gradient V_(F) in theX-t plane such as shown in FIG. 83. In other words, it can be said thatthe gradient V_(F) represents the direction of continuity in thetemporal and spatial directions in the X-t plane. Accordingly, the datacontinuity detecting unit 101 can output movement θ such as shown inFIG. 83 (strictly speaking, though not shown in the drawing, movement θis an angle generated by the direction of data continuity representedwith the gradient V_(f) corresponding to the gradient V_(F) and the Xdirection in the spatial direction) as data continuity informationcorresponding to the gradient V_(F) representing continuity in thetemporal and spatial directions in the X-t plane as well as the aboveangle θ (data continuity information corresponding to continuity in thespatial directions represented with the gradient G_(F) in the X-Yplane).

Accordingly, the actual world estimating unit 102 employing thetwo-dimensional polynomial approximating method can calculate thecoefficients (features) w_(i) of an approximation function f(x, t) inthe same method as the above method by employing the movement θ insteadof the angle θ. However, in this case, the equation to be employed isnot the above Expression (66) but the following Expression (82).

$\begin{matrix}{{P\left( {x,t} \right)} = {{\int_{t - 0.5}^{t + 0.5}{\int_{x - 0.5}^{x + 0.5}{\sum\limits_{i = 0}^{n}{{w_{i}\left( {x - {s \times t}} \right)}^{i}\ {\mathbb{d}x}\ {\mathbb{d}t}}}}} + e}} & (82)\end{matrix}$

Note that in Expression (82), s is cot θ (however, θ is movement).

Also, an approximation function f(y, t) focusing attention on thespatial direction Y instead of the spatial direction X can be handled inthe same way as the above approximation function f(x, t).

Thus, the two-dimensional polynomial approximating method takes notone-dimensional but two-dimensional integration effects intoconsideration, so can estimate the light signals in the actual world 1more accurately than the one-dimensional approximating method.

Next, description will be made regarding the third functionapproximating method with reference to FIG. 84 through FIG. 88.

That is to say, the third function approximating method is a method forestimating the light signal function F(x, y, t) by approximating thelight signal function F(x, y, t) with the approximation function f(x, y,t) focusing attention on that the light signal in the actual world 1having continuity in a predetermined direction of the temporal andspatial directions is represented with the light signal function F(x, y,t), for example. Accordingly, hereafter, the third functionapproximating method is referred to as a three-dimensional functionapproximating method.

Also, with description of the three-dimensional function approximatingmethod, let us say that the sensor 2 is a CCD made up of the multipledetecting elements 2-1 disposed on the plane thereof, such as shown inFIG. 84.

With the example in FIG. 84, the direction in parallel with apredetermined side of the detecting elements 2-1 is taken as the Xdirection serving as one direction of the spatial directions, and thedirection orthogonal to the X direction is taken as the Y directionserving as the other direction of the spatial directions. The directionorthogonal to the X-Y plane is taken as the t direction serving as thetemporal direction.

Also, with the example in FIG. 84, the spatial shape of the respectivedetecting elements 2-1 of the sensor 2 is taken as a square of which oneside is 1 in length. The shutter time (exposure time) of the sensor 2 istaken as 1.

Further, with the example in FIG. 84, the center of one certaindetecting element 2-1 of the sensor 2 is taken as the origin (theposition in the X direction is x=0, and the position in the Y directionis y=0) in the spatial directions (X direction and Y direction), andalso the intermediate point-in-time of the exposure time is taken as theorigin (the position in the t direction is t=0) in the temporaldirection (t direction).

In this case, the detecting element 2-1 of which the center is in theorigin (x=0, y=0) in the spatial directions subjects the light signalfunction F(x, y, t) to integration with a range of −0.5 through 0.5 inthe X direction, with a range of −0.5 through 0.5 in the Y direction,and with a range of −0.5 through 0.5 in the t direction, and outputs theintegral value as the pixel value P.

That is to say, the pixel value P output from the detecting element 2-1of which the center is in the origin in the spatial directions isrepresented with the following Expression (83).

$\begin{matrix}{P = {\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{{F\left( {x,y,t} \right)}\ {\mathbb{d}x}\ {\mathbb{d}y}\ {\mathbb{d}t}}}}}} & (83)\end{matrix}$

Similarly, the other detecting elements 2-1 output the pixel value Pshown in Expression (83) by taking the center of the detecting element2-1 to be processed as the origin in the spatial directions.

Incidentally, as described above, with the three-dimensional functionapproximating method, the light signal function F(x, y, t) isapproximated to the three-dimensional approximation function f(x, y, t).

Specifically, for example, the approximation function f(x, y, t) istaken as a function having N variables (features), a relationalexpression between the input pixel values P(x, y, t) corresponding toExpression (83) and the approximation function f(x, y, t) is defined.Thus, in the event that M input pixel values P(x, y, t) more than N areacquired, N variables (features) can be calculated from the definedrelational expression. That is to say, the actual world estimating unit102 can estimate the light signal function F(x, y, t) by acquiring Minput pixel values P(x, y, t), and calculating N variables (features).

In this case, the actual world estimating unit 102 extracts (acquires) Minput images P(x, y, t), of the entire input image by using continuityof data included in an input image (input pixel values) from the sensor2 as a constraint (i.e., using data continuity information as to aninput image to be output from the data continuity detecting unit 101).As a result, the approximation function f(x, y, t) is constrained bycontinuity of data.

For example, as shown in FIG. 85, in the event that the light signalfunction F(x, y, t) corresponding to an input image has continuity inthe spatial direction represented with the gradient G_(F), the datacontinuity detecting unit 101 results in outputting the angle θ (theangle θ generated between the direction of continuity of datarepresented with the gradient G_(f) (not shown) corresponding to thegradient G_(F), and the X direction) as data continuity information asto the input image.

In this case, let us say that a one-dimensional waveform wherein thelight signal function F(x, y, t) is projected in the X direction (such awaveform is referred to as an X cross-sectional waveform here) has thesame form even in the event of projection in any position in the Ydirection.

That is to say, let us say that there is an X cross-sectional waveformhaving the same form, which is a two-dimensional (spatial directional)waveform continuous in the direction of continuity (angle θ direction asto the X direction), and a three-dimensional waveform wherein such atwo-dimensional waveform continues in the temporal direction t, isapproximated with the approximation function f(x, y, t).

In other words, an X cross-sectional waveform, which is shifted by aposition y in the Y direction from the center of the pixel of interest,becomes a waveform wherein the X cross-sectional waveform passingthrough the center of the pixel of interest is moved (shifted) by apredetermined amount (amount varies according to the angle θ) in the Xdirection. Note that hereafter, such an amount is referred to as a shiftamount.

This shift amount can be calculated as follows.

That is to say, the gradient V_(f) (for example, gradient V_(f)representing the direction of data continuity corresponding to thegradient V_(F) in FIG. 85) and angle θ are represented as the followingExpression (84).

$\begin{matrix}{G_{f} = {{\tan\;\theta} = \frac{\mathbb{d}y}{\mathbb{d}x}}} & (84)\end{matrix}$

Note that in Expression (84), dx represents the amount of fine movementin the X direction, and dy represents the amount of fine movement in theY direction as to the dx.

Accordingly, if the shift amount as to the X direction is described asC_(x)(y), this is represented as the following Expression (85).

$\begin{matrix}{{C_{x}(y)} = \frac{y}{G_{f}}} & (85)\end{matrix}$

If the shift amount C_(x)(y) is thus defined, a relational expressionbetween the input pixel values P(x, y, t) corresponding to Expression(83) and the approximation function f(x, y, t) is represented as thefollowing Expression (86).

$\begin{matrix}{{P\left( {x,y,t} \right)} = {{\int_{t_{s}}^{t_{e}}{\int_{y_{s}}^{y_{e}}{\int_{x_{s}}^{x_{e}}{{f\left( {x,y,t} \right)}\ {\mathbb{d}x}\ {\mathbb{d}y}\ {\mathbb{d}t}}}}} + e}} & (86)\end{matrix}$

In Expression (86), e represents a margin of error. t_(s) represents anintegration start position in the t direction, and t_(e) represents anintegration end position in the t direction. In the same way, y_(s)represents an integration start position in the Y direction, and y_(e)represents an integration end position in the Y direction. Also, x_(s)represents an integration start position in the X direction, and x_(e)represents an integration end position in the X direction. However, therespective specific integral ranges are as shown in the followingExpression (87).

$\begin{matrix}{{t_{s} = {t - 0.5}}{t_{e} = {t + 0.5}}{y_{s} = {y - 0.5}}{y_{e} = {y + 0.5}}{x_{s} = {x - {C_{x}(y)} - 0.5}}{x_{e} = {x - {C_{x}(y)} + 0.5}}} & (87)\end{matrix}$

As shown in Expression (87), it can be represented that an Xcross-sectional waveform having the same form continues in the directionof continuity (angle θ direction as to the X direction) by shifting anintegral range in the X direction as to a pixel positioned distant fromthe pixel of interest by (x, y) in the spatial direction by the shiftamount C_(x)(y).

Thus, with the three-dimensional approximating method, the relationbetween the pixel values P(x, y, t) and the three-dimensionalapproximation function f(x, y, t) can be represented with Expression(86) (Expression (87) for the integral range), and accordingly, thelight signal function F(x, y, t) (for example, a light signal havingcontinuity in the spatial direction represented with the gradient V_(F)such as shown in FIG. 85) can be estimated by calculating the N featuresof the approximation function f(x, y, t), for example, with the leastsquare method using Expression (86) and Expression (87).

Note that in the event that a light signal represented with the lightsignal function F(x, y, t) has continuity in the spatial directionrepresented with the gradient V_(F) such as shown in FIG. 85, the lightsignal function F(x, y, t) may be approximated as follows.

That is to say, let us say that a one-dimensional waveform wherein thelight signal function F(x, y, t) is projected in the Y direction(hereafter, such a waveform is referred to as a Y cross-sectionalwaveform) has the same form even in the event of projection in anyposition in the X direction.

In other words, let us say that there is a two-dimensional (spatialdirectional) waveform wherein a Y cross-sectional waveform having thesame form continues in the direction of continuity (angle θ direction asto in the X direction), and a three-dimensional waveform wherein such atwo-dimensional waveform continues in the temporal direction t isapproximated with the approximation function f(x, y, t).

Accordingly, the Y cross-sectional waveform, which is shifted by x inthe X direction from the center of the pixel of interest, becomes awaveform wherein the Y cross-sectional waveform passing through thecenter of the pixel of interest is moved by a predetermined shift amount(shift amount changing according to the angle θ) in the Y direction.

This shift amount can be calculated as follows.

That is to say, the gradient G_(F) is represented as the aboveExpression (84), so if the shift amount as to the Y direction isdescribed as C_(y)(x), this is represented as the following Expression(88).

$\begin{matrix}{{C_{y}(x)} = {G_{f} \times x}} & (88)\end{matrix}$

If the shift amount C_(x)(y) is thus defined, a relational expressionbetween the input pixel values P(x, y, t) corresponding to Expression(83) and the approximation function f(x, y, t) is represented as theabove Expression (86), as with when the shift amount C_(x)(y) isdefined.

However, in this case, the respective specific integral ranges are asshown in the following Expression (89).

$\begin{matrix}{{t_{s} = {t - 0.5}}{t_{e} = {t + 0.5}}{y_{s} = {y - {C_{y}(x)} - 0.5}}{y_{e} = {y - {C_{y}(x)} + 0.5}}{x_{s} = {x - 0.5}}{x_{e} = {x + 0.5}}} & (89)\end{matrix}$

As shown in Expression (89) (and the above Expression (86)), it can berepresented that a Y cross-sectional waveform having the same formcontinues in the direction of continuity (angle θ direction as to the Xdirection) by shifting an integral range in the Y direction as to apixel positioned distant from the pixel of interest by (x, y), by theshift amount C_(x)(y).

Thus, with the three-dimensional approximating method, the integralrange of the right side of the above Expression (86) can be set to notonly Expression (87) but also Expression (89), and accordingly, thelight signal function F(x, y, t) (light signal in the actual world 1having continuity in the spatial direction represented with the gradientG_(F)) can be estimated by calculating the n features of theapproximation function f(x, y, t) with, for example, the least squaremethod or the like using Expression (86) in which Expression (89) isemployed as an integral range.

Thus, Expression (87) and Expression (89), which represent an integralrange, represent essentially the same with only a difference regardingwhether perimeter pixels are shifted in the X direction (in the case ofExpression (87)) or shifted in the Y direction (in the case ofExpression (89)) in response to the direction of continuity.

However, in response to the direction of continuity (gradient G_(F)),there is a difference regarding whether the light signal function F(x,y, t) is regarded as a group of X cross-sectional waveforms, or isregarded as a group of Y cross-sectional waveforms. That is to say, inthe event that the direction of continuity is close to the Y direction,the light signal function F(x, y, t) is preferably regarded as a groupof X cross-sectional waveforms. On the other hand, in the event that thedirection of continuity is close to the X direction, the light signalfunction F(x, y, t) is preferably regarded as a group of Ycross-sectional waveforms.

Accordingly, it is preferable that the actual world estimating unit 102prepares both Expression (87) and Expression (89) as an integral range,and selects any one of Expression (87) and Expression (89) as theintegral range of the right side of the appropriate Expression (86) inresponse to the direction of continuity.

Description has been made regarding the three-dimensional approximationmethod in the case in which the light signal function F(x, y, t) hascontinuity (for example, continuity in the spatial direction representedwith the gradient G_(F) in FIG. 85) in the spatial directions (Xdirection and Y direction), but the three-dimensional approximationmethod can be applied to the case in which the light signal functionF(x, y, t) has continuity (continuity represented with the gradientV_(F)) in the temporal and spatial directions (X direction, Y direction,and t direction), as shown in FIG. 86.

That is to say, in FIG. 86, a light signal function corresponding to aframe #N−1 is taken as F(x, y, #N−1), a light signal functioncorresponding to a frame #N is taken as F(x, y, #N), and a light signalfunction corresponding to a frame #N+1 is taken as F(x, y, #N+1).

Note that in FIG. 86, the horizontal direction is taken as the Xdirection serving as one direction of the spatial directions, the upperright diagonal direction is taken as the Y direction serving as theother direction of the spatial directions, and also the verticaldirection is taken as the t direction serving as the temporal directionin the drawing.

Also, the frame #N−1 is a frame temporally prior to the frame #N, theframe #N+1 is a frame temporally following the frame #N. That is to say,the frame #N−1, frame #N, and frame #N+1 are displayed in the sequenceof the frame #N−1, frame #N, and frame #N+1.

With the example in FIG. 86, a cross-sectional light level along thedirection shown with the gradient V_(F) (upper right inner directionfrom lower left near side in the drawing) is regarded as generallyconstant. Accordingly, with the example in FIG. 86, it can be said thatthe light signal function F(x, y, t) has continuity in the temporal andspatial directions represented with the gradient V_(F).

In this case, in the event that a function C(x, y, t) representingcontinuity in the temporal and spatial directions is defined, and alsothe integral range of the above Expression (86) is defined with thedefined function C(x, y, t), N features of the approximation functionf(x, y, t) can be calculated as with the above Expression (87) andExpression (89).

The function C(x, y, t) is not restricted to a particular function aslong as this is a function representing the direction of continuity.However, hereafter, let us say that linear continuity is employed, andC_(x)(t) and C_(y)(t) corresponding to the shift amount C_(x)(y)(Expression (85)) and shift amount C_(y)(x) (Expression (87)), which arefunctions representing continuity in the spatial direction describedabove, are defined as a function C(x, y, t) corresponding thereto asfollows.

That is to say, if the gradient as continuity of data in the temporaland spatial directions corresponding to the gradient G_(f) representingcontinuity of data in the above spatial direction is taken as V_(f), andif this gradient V_(f) is divided into the gradient in the X direction(hereafter, referred to as V_(fx)) and the gradient in the Y direction(hereafter, referred to as V_(fy)), the gradient V_(fx) is representedwith the following Expression (90), and the gradient V_(fy) isrepresented with the following Expression (91), respectively.

$\begin{matrix}{V_{fx} = \frac{\mathbb{d}x}{\mathbb{d}t}} & (90) \\{V_{fy} = \frac{\mathbb{d}y}{\mathbb{d}t}} & (91)\end{matrix}$

In this case, the function C_(x)(t) is represented as the followingExpression (92) using the gradient V_(fx) shown in Expression (90).

$\begin{matrix}{{C_{x}(t)} = {V_{fx} \times t}} & (92)\end{matrix}$

Similarly, the function C_(y)(t) is represented as the followingExpression (93) using the gradient V_(fy) shown in Expression (91).

$\begin{matrix}{{C_{y}(t)} = {V_{fy} \times t}} & (93)\end{matrix}$

Thus, upon the function C_(x)(t) and function C_(y)(t), which representcontinuity 2511 in the temporal and spatial directions, being defined,the integral range of Expression (86) is represented as the followingExpression (94).

$\begin{matrix}{{t_{s} = {t - 0.5}}{t_{e} = {t + 0.5}}{y_{s} = {y - {C_{y}(t)} - 0.5}}{y_{e} = {y - {C_{y}(t)} + 0.5}}{x_{s} = {x - {C_{x}(t)} - 0.5}}{x_{e} = {x - {C_{x}(t)} + 0.5}}} & (94)\end{matrix}$

Thus, with the three-dimensional approximating method, the relationbetween the pixel values P(x, y, t) and the three-dimensionalapproximation function f(x, y, t) can be represented with Expression(86), and accordingly, the light signal function F(x, y, t) (lightsignal in the actual world 1 having continuity in a predetermineddirection of the temporal and spatial directions) can be estimated bycalculating the n+1 features of the approximation function f(x, y, t)with, for example, the least square method or the like using Expression(94) as the integral range of the right side of Expression (86).

FIG. 87 represents a configuration example of the actual worldestimating unit 102 employing such a three-dimensional approximatingmethod.

Note that the approximation function f(x, y, t) (in reality, thefeatures (coefficients) thereof) calculated by the actual worldestimating unit 102 employing the three-dimensional approximating methodis not restricted to a particular function, but an n (n=N−1)-dimensionalpolynomial is employed in the following description.

As shown in FIG. 87, the actual world estimating unit 102 includes aconditions setting unit 2521, input image storage unit 2522, input pixelvalue acquiring unit 2523, integral component calculation unit 2524,normal equation generating unit 2525, and approximation functiongenerating unit 2526.

The conditions setting unit 2521 sets a pixel range (tap range) used forestimating the light signal function F(x, y, t) corresponding to a pixelof interest, and the number of dimensions n of the approximationfunction f(x, y, t).

The input image storage unit 2522 temporarily stores an input image(pixel values) from the sensor 2.

The input pixel acquiring unit 2523 acquires, of the input images storedin the input image storage unit 2522, an input image regioncorresponding to the tap range set by the conditions setting unit 2521,and supplies this to the normal equation generating unit 2525 as aninput pixel value table. That is to say, the input pixel value table isa table in which the respective pixel values of pixels included in theinput image region are described.

Incidentally, as described above, the actual world estimating unit 102employing the three-dimensional approximating method calculates the Nfeatures (in this case, coefficient of each dimension) of theapproximation function f(x, y, t) with the least square method using theabove Expression (86) (however, Expression (87), Expression (90), orExpression (94) for the integral range).

The right side of Expression (86) can be represented as the followingExpression (95) by calculating the integration thereof.

$\begin{matrix}{{P\left( {x,y,t} \right)} = {{\sum\limits_{i = 0}^{n}{w_{i}{S_{i}\left( {x_{s},x_{e},y_{s},y_{e},t_{s},t_{e}} \right)}}} + e}} & (95)\end{matrix}$

In Expression (95), w_(i) represents the coefficients (features) of thei-dimensional term, and also S_(i)(x_(s), x_(e), y_(s), y_(e), t_(s),t_(e)) represents the integral components of the i-dimensional term.However, x_(s) represents an integral range start position in the Xdirection, x_(e) represents an integral range end position in the Xdirection, y_(s) represents an integral range start position in the Ydirection, y_(e) represents an integral range end position in the Ydirection, t_(s) represents an integral range start position in the tdirection, t_(e) represents an integral range end position in the tdirection, respectively.

The integral component calculation unit 2524 calculates the integralcomponents S_(i)(x_(s), x_(e), y_(s), y_(e), t_(s), t_(e)).

That is to say, the integral component calculation unit 2524 calculatesthe integral components S_(i)(x_(s), x_(e), y_(s), y_(e), t_(s), t_(e))based on the tap range and the number of dimensions set by theconditions setting unit 2521, and the angle or movement (as the integralrange, angle in the case of using the above Expression (87) orExpression (90), and movement in the case of using the above Expression(94)) of the data continuity information output from the data continuitydetecting unit 101, and supplies the calculated results to the normalequation generating unit 2525 as an integral component table.

The normal equation generating unit 2525 generates a normal equation inthe case of obtaining the above Expression (95) with the least squaremethod using the input pixel value table supplied from the input pixelvalue acquiring unit 2523, and the integral component table suppliedfrom the integral component calculation unit 2524, and outputs this tothe approximation function generating unit 2526 as a normal equationtable. An example of a normal equation will be described later.

The approximation function generating unit 2526 calculates therespective features w_(i) (in this case, the coefficients w_(i) of theapproximation function f(x, y, t) serving as a three-dimensionalpolynomial) by solving the normal equation included in the normalequation table supplied from the normal equation generating unit 2525with the matrix solution, and output these to the image generating unit103.

Next, description will be made regarding the actual world estimatingprocessing (processing in step S102 in FIG. 29) to which thethree-dimensional approximating method is applied, with reference to theflowchart in FIG. 88.

First, in step S2501, the conditions setting unit 2521 sets conditions(a tap range and the number of dimensions).

For example, let us say that a tap range made up of L pixels has beenset. Also, let us say that a predetermined number l (l is any one ofinteger values 0 through L−1) is appended to each of the pixels.

Next, in step S2502, the conditions setting unit 2521 sets a pixel ofinterest.

In step S2503, the input pixel value acquiring unit 2523 acquires aninput pixel value based on the condition (tap range) set by theconditions setting unit 2521, and generates an input pixel value table.In this case, a table made up of L input pixel values P(x, y, t) isgenerated. Here, let us say that each of the L input pixel values P(x,y, t) is described as P(l) serving as a function of the number l of thepixel thereof. That is to say, the input pixel value table becomes atable including L P(l).

In step S2504, the integral component calculation unit 2524 calculatesintegral components based on the conditions (a tap range and the numberof dimensions) set by the conditions setting unit 2521, and the datacontinuity information (angle or movement) supplied from the datacontinuity detecting unit 101, and generates an integral componenttable.

However, in this case, as described above, the input pixel values arenot P(x, y, t) but P(l), and are acquired as the value of a pixel numberl, so the integral component calculation unit 2524 results incalculating the integral components S_(i)(x_(s), x_(e), y_(s), y_(e),t_(s), t_(e)) in the above Expression (95) as a function of l such asthe integral components S_(i)(l). That is to say, the integral componenttable becomes a table including L×i S_(i)(l).

Note that the sequence of the processing in step S2503 and theprocessing in step S2504 is not restricted to the example in FIG. 88, sothe processing in step S2504 may be executed first, or the processing instep S2503 and the processing in step S2504 may be executedsimultaneously.

Next, in step S2505, the normal equation generating unit 2525 generatesa normal equation table based on the input pixel value table generatedby the input pixel value acquiring unit 2523 at the processing in step2503, and the integral component table generated by the integralcomponent calculation unit 2524 at the processing in step S2504.

Specifically, in this case, the features w_(i) of the followingExpression (96) corresponding to the above Expression (95) arecalculated using the least square method. A normal equationcorresponding to this is represented as the following Expression (97).

$\begin{matrix}{{P(l)} = {{\sum\limits_{i = 0}^{n}{w_{i}{S_{i}(l)}}} + e}} & (96) \\{{\begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}\begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (97)\end{matrix}$

If we define each matrix of the normal equation shown in Expression (97)as the following Expressions (98) through (100), the normal equation isrepresented as the following Expression (101).

$\begin{matrix}{S_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{0}(l)}{S_{n}(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{1}(l)}{S_{n}(l)}}} \\\vdots & \vdots & ⋰ & \vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{0}(l)}}} & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{1}(l)}}} & \ldots & {\sum\limits_{l = 0}^{L}{{S_{n}(l)}{S_{n}(l)}}}\end{pmatrix}} & (98) \\{W_{MAT} = \begin{pmatrix}w_{0} \\w_{1} \\\vdots \\w_{n}\end{pmatrix}} & (99) \\{P_{MAT} = \begin{pmatrix}{\sum\limits_{l = 0}^{L}{{S_{0}(l)}{P(l)}}} \\{\sum\limits_{l = 0}^{L}{{S_{1}(l)}{P(l)}}} \\\vdots \\{\sum\limits_{l = 0}^{L}{{S_{n}(l)}{P(l)}}}\end{pmatrix}} & (100) \\{{S_{MAT}W_{MAT}} = P_{MAT}} & (101)\end{matrix}$

As shown in Expression (99), the respective components of the matrixW_(MAT) are the features w_(i) to be obtained. Accordingly, inExpression (101), if the matrix S_(MAT) of the left side and the matrixP_(MAT) of the right side are determined, the matrix W_(MAT) (i.e.,features w_(i)) may be calculated with the matrix solution.

Specifically, as shown in Expression (98), the respective components ofthe matrix S_(MAT) may be calculated as long as the above integralcomponents S_(i)(l) are known. The integral components S_(i)(l) areincluded in the integral component table supplied from the integralcomponent calculation unit 2524, so the normal equation generating unit2525 can calculate each component of the matrix S_(MAT) using theintegral component table.

Also, as shown in Expression (100), the respective components of thematrix P_(MAT) may be calculated as long as the integral componentsS_(i)(l) and the input pixel values P(l) are known. The integralcomponents S_(i)(l) is the same as those included in the respectivecomponents of the matrix S_(MAT), also the input pixel values P(l) areincluded in the input pixel value table supplied from the input pixelvalue acquiring unit 2523, so the normal equation generating unit 2525can calculate each component of the matrix P_(MAT) using the integralcomponent table and input pixel value table.

Thus, the normal equation generating unit 2525 calculates each componentof the matrix S_(MAT) and matrix P_(MAT), and outputs the calculatedresults (each component of the matrix S_(MAT) and matrix P_(MAT)) to theapproximation function generating unit 2526 as a normal equation table.

Upon the normal equation table being output from the normal equationgenerating unit 2526, in step S2506, the approximation functiongenerating unit 2526 calculates the features w_(i) (i.e., thecoefficients w_(i) of the approximation function f(x, y, t)) serving asthe respective components of the matrix W_(MAT) in the above Expression(101) based on the normal equation table.

Specifically, the normal equation in the above Expression (101) can betransformed as the following Expression (102).

$\begin{matrix}{W_{MAT} = {S_{MAT}^{- 1}P_{MAT}}} & (102)\end{matrix}$

In Expression (102), the respective components of the matrix W_(MAT) inthe left side are the features w_(i) to be obtained. The respectivecomponents regarding the matrix S_(MAT) and matrix P_(MAT) are includedin the normal equation table supplied from the normal equationgenerating unit 2525. Accordingly, the approximation function generatingunit 2526 calculates the matrix W_(MAT) by calculating the matrix in theright side of Expression (102) using the normal equation table, andoutputs the calculated results (features w_(i)) to the image generatingunit 103.

In step S2507, the approximation function generating unit 2526determines regarding whether or not the processing of all the pixels hasbeen completed.

In step S2507, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S2502, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S2502 through S2507 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S2507, in the event that determination is made that theprocessing of all the pixels has been completed), the estimatingprocessing of the actual world 1 ends.

As described above, the three-dimensional approximating method takesthree-dimensional integration effects in the temporal and spatialdirections into consideration instead of one-dimensional ortwo-dimensional integration effects, and accordingly, can estimate thelight signals in the actual world 1 more accurately than theone-dimensional approximating method and two-dimensional polynomialapproximating method.

Next, description will be made regarding an embodiment of the imagegenerating unit 103 (FIG. 3) with reference to FIG. 89 through FIG. 110.

FIG. 89 is a diagram for describing the principle of the presentembodiment.

As shown in FIG. 89, the present embodiment is based on condition thatthe actual world estimating unit 102 employs a function approximatingmethod. That is to say, let us say that the signals in the actual world1 (distribution of light intensity) serving as an image cast in thesensor 2 are represented with a predetermined function F, it is anassumption for the actual world estimating unit 102 to estimate thefunction F by approximating the function F with a predetermined functionf using the input image (pixel value P) output from the sensor 2 and thedata continuity information output from the data continuity detectingunit 101.

Note that hereafter, with description of the present embodiment, thesignals in the actual world 1 serving as an image are particularlyreferred to as light signals, and the function F is particularlyreferred to as a light signal function F. Also, the function f isparticularly referred to as an approximation function f.

With the present embodiment, the image generating unit 103 integratesthe approximation function f with a predetermined time-space range usingthe data continuity information output from the data continuitydetecting unit 101, and the actual world estimating information (in theexample in FIG. 89, the features of the approximation function f orapproximation function f of which the features are identified) outputfrom the actual world estimating unit 102 based on such an assumption,and outputs the integral value as an output pixel value M (outputimage). Note that with the present embodiment, an input pixel value isdescribed as P, and an output pixel value is described as M in order todistinguish an input image pixel from an output image pixel.

In other words, upon the light signal function F being integrated once,the light signal function F becomes an input pixel value P, the lightsignal function F is estimated from the input pixel value P(approximated with the approximation function f), the estimated lightsignal function F (i.e., approximation function f) is integrated again,and an output pixel value M is generated. Accordingly, hereafter,integration of the approximation function f executed by the imagegenerating unit 103 is referred to as reintegration. Also, the presentembodiment is referred to as a reintegration method.

Note that as described later, with the reintegration method, theintegral range of the approximation function f in the event that theoutput pixel value M is generated is not restricted to the integralrange of the light signal function F in the event that the input pixelvalue P is generated (i.e., the vertical width and horizontal width ofthe detecting element of the sensor 2 for the spatial direction, theexposure time of the sensor 2 for the temporal direction), an arbitraryintegral range may be employed.

For example, in the event that the output pixel value M is generated,varying the integral range in the spatial direction of the integralrange of the approximation function f enables the pixel pitch of anoutput image according to the integral range thereof to be varied. Thatis to say, creation of spatial resolution is available.

In the same way, for example, in the event that the output pixel value Mis generated, varying the integral range in the temporal direction ofthe integral range of the approximation function f enables creation oftemporal resolution.

Hereafter, description will be made individually regarding threespecific methods of such a reintegration method with reference to thedrawings.

That is to say, three specific methods are reintegration methodscorresponding to three specific methods of the function approximatingmethod (the above three specific examples of the embodiment of theactual world estimating unit 102) respectively.

Specifically, the first method is a reintegration method correspondingto the above one-dimensional approximating method (one method of thefunction approximating method). Accordingly, with the first method,one-dimensional reintegration is performed, so hereafter, such areintegration method is referred to as a one-dimensional reintegrationmethod.

The second method is a reintegration method corresponding to the abovetwo-dimensional polynomial approximating method (one method of thefunction approximating method). Accordingly, with the second method,two-dimensional reintegration is performed, so hereafter, such areintegration method is referred to as a two-dimensional reintegrationmethod.

The third method is a reintegration method corresponding to the abovethree-dimensional approximating method (one method of the functionapproximating method). Accordingly, with the third method,three-dimensional reintegration is performed, so hereafter, such areintegration method is referred to as a three-dimensional reintegrationmethod.

Hereafter, description will be made regarding each details of theone-dimensional reintegration method, two-dimensional reintegrationmethod, and three-dimensional reintegration method in this order.

First, the one-dimensional reintegration method will be described.

With the one-dimensional reintegration method, it is assumed that theapproximation function f(x) is generated using the one-dimensionalapproximating method.

That is to say, it is an assumption that a one-dimensional waveform(with description of the reintegration method, a waveform projected inthe X direction of such a waveform is referred to as an Xcross-sectional waveform F(x)) wherein the light signal function F(x, y,t) of which variables are positions x, y, and z on the three-dimensionalspace, and a point-in-time t is projected in a predetermined direction(for example, X direction) of the X direction, Y direction, and Zdirection serving as the spatial direction, and t direction serving asthe temporal direction, is approximated with the approximation functionf(x) serving as an n-dimensional (n is an arbitrary integer) polynomial.

In this case, with the one-dimensional reintegration method, the outputpixel value M is calculated such as the following Expression (103).

$\begin{matrix}{M = {G_{e} \times {\int_{x_{s}}^{x_{e}}{{f(x)}\ {\mathbb{d}x}}}}} & (103)\end{matrix}$

Note that in Expression (103), x_(s) represents an integration startposition, x_(e) represents an integration end position. Also, G_(e)represents a predetermined gain.

Specifically, for example, let us say that the actual world estimatingunit 102 has already generated the approximation function f(x) (theapproximation function f(x) of the X cross-sectional waveform F(x)) suchas shown in FIG. 90 with a pixel 3101 (pixel 3101 corresponding to apredetermined detecting element of the sensor 2) such as shown in FIG.90 as a pixel of interest.

Note that with the example in FIG. 90, the pixel value (input pixelvalue) of the pixel 3101 is taken as P, and the shape of the pixel 3101is taken as a square of which one side is 1 in length. Also, of thespatial directions, the direction in parallel with one side of the pixel3101 (horizontal direction in the drawing) is taken as the X direction,and the direction orthogonal to the X direction (vertical direction inthe drawing) is taken as the Y direction.

Also, on the lower side in FIG. 90, the coordinates system (hereafter,referred to as a pixel-of-interest coordinates system) in the spatialdirections (X direction and Y direction) of which the center of thepixel 3101 is taken as the origin, and the pixel 3101 in the coordinatessystem are shown.

Further, on the upward direction in FIG. 90, a graph representing theapproximation function f(x) at y=0 (y is a coordinate value in the Ydirection in the pixel-of-interest coordinates system shown on the lowerside in the drawing) is shown. In this graph, the axis in parallel withthe horizontal direction in the drawing is the same axis as the x axisin the X direction in the pixel-of-interest coordinates system shown onthe lower side in the drawing (the origin is also the same), and alsothe axis in parallel with the vertical direction in the drawing is takenas an axis representing pixel values.

In this case, the relation of the following Expression (104) holdsbetween the approximation function f(x) and the pixel value P of thepixel 3101.

$\begin{matrix}{P = {{\int_{- 0.5}^{0.5}{{f(x)}\ {\mathbb{d}x}}} + e}} & (104)\end{matrix}$

Also, as shown in FIG. 90, let us say that the pixel 3101 has continuityof data in the spatial direction represented with the gradient G_(f).Further, let us say that the data continuity detecting unit 101 (FIG.89) has already output the angle θ such as shown in FIG. 90 as datacontinuity information corresponding to continuity of data representedwith the gradient G_(f).

In this case, for example, with the one-dimensional reintegrationmethod, as shown in FIG. 91, four pixels 3111 through 3114 can be newlycreated in a range of −0.5 through 0.5 in the X direction, and also in arange of −0.5 through 0.5 in the Y direction (in the range where thepixel 3101 in FIG. 90 is positioned).

Note that on the lower side in FIG. 91, the same pixel-of-interestcoordinates system as that in FIG. 90, and the pixels 3111 through 3114in the pixel-of-interest coordinates system thereof are shown. Also, onthe upper side in FIG. 91, the same graph (graph representing theapproximation function f(x) at y=0) as that in FIG. 90 is shown.

Specifically, as shown in FIG. 91, with the one-dimensionalreintegration method, calculation of the pixel value M(1) of the pixel3111 using the following Expression (105), calculation of the pixelvalue M(2) of the pixel 3112 using the following Expression (106),calculation of the pixel value M(3) of the pixel 3113 using thefollowing Expression (107), and calculation of the pixel value M(4) ofthe pixel 3114 using the following Expression (108) are availablerespectively.

$\begin{matrix}{{M(1)} = {2 \times {\int_{x_{s\; 1}}^{x_{e\; 1}}{{f(x)}{\mathbb{d}x}}}}} & (105) \\{{M(2)} = {2 \times {\int_{x_{s\; 2}}^{x_{e\; 2}}{{f(x)}{\mathbb{d}x}}}}} & (106) \\{{M(3)} = {2 \times {\int_{x_{s\; 3}}^{x_{e\; 3}}{{f(x)}{\mathbb{d}x}}}}} & (107) \\{{M(4)} = {2 \times {\int_{x_{s\; 4}}^{x_{e\; 4}}{{f(x)}{\mathbb{d}x}}}}} & (108)\end{matrix}$

Note that x_(s1) in Expression (105), x_(s2) in Expression (106), x_(s3)in Expression (107), and x_(s4) in Expression (108) each represent theintegration start position of the corresponding expression. Also, x_(e1)in Expression (105), x_(e2) in Expression (106), x_(e3) in Expression(107), and x_(e4) in Expression (108) each represent the integration endposition of the corresponding expression.

The integral range in the right side of each of Expression (105) throughExpression (108) becomes the pixel width (length in the X direction) ofeach of the pixel 3111 through pixel 3114. That is to say, each ofx_(e1)−x_(s1), x_(e2)−x_(s2), x_(e3)−x_(s3), and x_(e4)−x_(s4) becomes0.5.

However, in this case, it can be conceived that a one-dimensionalwaveform having the same form as that in the approximation function f(x)at y=0 continues not in the Y direction but in the direction of datacontinuity represented with the gradient G_(f) (i.e., angle θ direction)(in fact, a waveform having the same form as the X cross-sectionalwaveform F(x) at y=0 continues in the direction of continuity). That isto say, in the case in which a pixel value f (0) in the origin (0, 0) inthe pixel-of-interest coordinates system in FIG. 91 (center of the pixel3101 in FIG. 90) is taken as a pixel value f1, the direction where thepixel value f1 continues is not the Y direction but the direction ofdata continuity represented with the gradient G_(f) (angle θ direction).

In other words, in the case of conceiving the waveform of theapproximation function f(x) in a predetermined position y in the Ydirection (however, y is a numeric value other than zero), the positioncorresponding to the pixel value f1 is not a position (0, y) but aposition (C_(x)(y), y) obtained by moving in the X direction from theposition (0, y) by a predetermined amount (here, let us say that such anamount is also referred to as a shift amount. Also, a shift amount is anamount depending on the position y in the Y direction, so let us saythat this shift amount is described as C_(x)(y)) Accordingly, as theintegral range of the right side of each of the above Expression (105)through Expression (108), the integral range needs to be set in light ofthe position y in the Y direction where the center of the pixel valueM(l) to be obtained (however, l is any integer value of 1 through 4)exists, i.e., the shift amount C_(x)(y).

Specifically, for example, the position y in the Y direction where thecenters of the pixel 3111 and pixel 3112 exist is not y=0 but y=0.25.

Accordingly, the waveform of the approximation function f(x) at y=0.25is equivalent to a waveform obtained by moving the waveform of theapproximation function f(x) at y=0 by the shift amount C_(x)(0.25) inthe X direction.

In other words, in the above Expression (105), if we say that the pixelvalue M(1) as to the pixel 3111 is obtained by integrating theapproximation function f(x) at y=0 with a predetermined integral range(from the start position x_(s1) to the end position x_(e1)), theintegral range thereof becomes not a range from the start positionx_(s1)=−0.5 to the end position x_(e1)=0 (a range itself where the pixel3111 occupies in the X direction) but the range shown in FIG. 91, i.e.,from the start position x_(s1)=−0.5+C_(x)(0.25) to the end positionx_(e1)=0+C_(x)(0.25) (a range where the pixel 3111 occupies in the Xdirection in the event that the pixel 3111 is tentatively moved by theshift amount C_(x)(0.25)).

Similarly, in the above Expression (106), if we say that the pixel valueM(2) as to the pixel 3112 is obtained by integrating the approximationfunction f(x) at y=0 with a predetermined integral range (from the startposition x_(s2) to the end position x_(e2)), the integral range thereofbecomes not a range from the start position x_(s2)=0 to the end positionx_(e2)=0.5 (a range itself where the pixel 3112 occupies in the Xdirection) but the range shown in FIG. 91, i.e., from the start positionx_(s2)=0+C_(x)(0.25) to the end position x_(e1)=0.5+C_(x)(0.25) (a rangewhere the pixel 3112 occupies in the X direction in the event that thepixel 3112 is tentatively moved by the shift amount C_(x)(0.25)).

Also, for example, the position y in the Y direction where the centersof the pixel 3113 and pixel 3114 exist is not y=0 but y=−0.25.

Accordingly, the waveform of the approximation function f(x) at y=−0.25is equivalent to a waveform obtained by moving the waveform of theapproximation function f(x) at y =0 by the shift amount C_(x)(−0.25) inthe X direction.

In other words, in the above Expression (107), if we say that the pixelvalue M(3) as to the pixel 3113 is obtained by integrating theapproximation function f(x) at y =0 with a predetermined integral range(from the start position x_(s3) to the end position x_(e3)), theintegral range thereof becomes not a range from the start positionx_(s3)=−0.5 to the end position x_(e3)=0 (a range itself where the pixel3113 occupies in the X direction) but the range shown in FIG. 91, i.e.,from the start position x_(s3)=−0.5+C_(x)(−0.25) to the end positionx_(e3)=0+C_(x)(−0.25) (a range where the pixel 3113 occupies in the Xdirection in the event that the pixel 3113 is tentatively moved by theshift amount C_(x)(−0.25)).

Similarly, in the above Expression (108), if we say that the pixel valueM(4) as to the pixel 3114 is obtained by integrating the approximationfunction f(x) at y=0 with a predetermined integral range (from the startposition x_(s4) to the end position x_(e4)), the integral range thereofbecomes not a range from the start position x_(s4)=0 to the end positionx_(e4)=0.5 (a range itself where the pixel 3114 occupies in the Xdirection) but the range shown in FIG. 91, i.e., from the start positionx_(s4)=0+C_(x)(−0.25) to the end position x_(e1)=0.5+C_(x)(−0.25) (arange where the pixel 3114 occupies in the X direction in the event thatthe pixel 3114 is tentatively moved by the shift amount C_(x)(−0.25)).

Accordingly, the image generating unit 102 (FIG. 89) calculates theabove Expression (105) through Expression (108) by substituting thecorresponding integral range of the above integral ranges for each ofthese expressions, and outputs the calculated results of these as theoutput pixel values M(1) through M(4).

Thus, the image generating unit 102 can create four pixels having higherspatial resolution than that of the output pixel 3101, i.e., the pixel3111 through pixel 3114 (FIG. 91) by employing the one-dimensionalreintegration method as a pixel at the output pixel 3101 (FIG. 90) fromthe sensor 2 (FIG. 89). Further, though not shown in the drawing, asdescribed above, the image generating unit 102 can create a pixel havingan arbitrary powered spatial resolution as to the output pixel 3101without deterioration by appropriately changing an integral range, inaddition to the pixel 3111 through pixel 3114.

FIG. 92 represents a configuration example of the image generating unit103 employing such a one-dimensional reintegration method.

As shown in FIG. 92, the image generating unit 103 shown in this exampleincludes a conditions setting unit 3121, features storage unit 3122,integral component calculation unit 3123, and output pixel valuecalculation unit 3124.

The conditions setting unit 3121 sets the number of dimensions n of theapproximation function f(x) based on the actual world estimatinginformation (the features of the approximation function f(x) in theexample in FIG. 92) supplied from the actual world estimating unit 102.

The conditions setting unit 3121 also sets an integral range in the caseof reintegrating the approximation function f(x) (in the case ofcalculating an output pixel value). Note that an integral range set bythe conditions setting unit 3121 does not need to be the width of apixel. For example, the approximation function f(x) is integrated in thespatial direction (X direction), and accordingly, a specific integralrange can be determined as long as the relative size (power of spatialresolution) of an output pixel (pixel to be calculated by the imagegenerating unit 103) as to the spatial size of each pixel of an inputimage from the sensor 2 (FIG. 89) is known. Accordingly, the conditionssetting unit 3121 can set, for example, a spatial resolution power as anintegral range.

The features storage unit 3122 temporally stores the features of theapproximation function f(x) sequentially supplied from the actual worldestimating unit 102. Subsequently, upon the features storage unit 3122storing all of the features of the approximation function f(x), thefeatures storage unit 3122 generates a features table including all ofthe features of the approximation function f(x), and supplies this tothe output pixel value calculation unit 3124.

Incidentally, as described above, the image generating unit 103calculates the output pixel value M using the above Expression (103),but the approximation function f(x) included in the right side of theabove Expression (103) is represented as the following Expression (109)specifically.

$\begin{matrix}{{f(x)} = {\sum\limits_{i = 0}^{n}{w_{i} \times x^{i}d\; x}}} & (109)\end{matrix}$

Note that in Expression (109), w_(i) represents the features of theapproximation function f(x) supplied from the actual world estimatingunit 102.

Accordingly, upon the approximation function f(x) of Expression (109)being substituted for the approximation function f(x) of the right sideof the above Expression (103) so as to expand (calculate) the right sideof Expression (103), the output pixel value M is represented as thefollowing Expression (110).

$\begin{matrix}\begin{matrix}{M = {G_{e} \times {\sum\limits_{i = 0}^{n}{w_{i} \times \frac{x_{e}^{i + 1} - x_{s}^{i + 1}}{i + 1}}}}} \\{= {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}\left( {x_{s},x_{e}} \right)}}}}\end{matrix} & (110)\end{matrix}$

In Expression (110), K_(i)(x_(s), x_(e)) represent the integralcomponents of the i-dimensional term. That is to say, the integralcomponents K_(i)(x_(s), x_(e)) are such as shown in the followingExpression (111).

$\begin{matrix}{{k_{i}\left( {x_{s},x_{e}} \right)} = {G_{e} \times \frac{x_{e}^{i + 1} - x_{s}^{i + 1}}{i + 1}}} & (111)\end{matrix}$

The integral component calculation unit 3123 calculates the integralcomponents K_(i)(x_(s), x_(e)).

Specifically, as shown in Expression (111), the integral componentsK_(i)(x_(s), x_(e)) can be calculated as long as the start positionx_(s) and end position x_(e) of an integral range, gain G_(e), and i ofthe i-dimensional term are known.

Of these, the gain G_(e) is determined with the spatial resolution power(integral range) set by the conditions setting unit 3121.

The range of i is determined with the number of dimensions n set by theconditions setting unit 3121.

Also, each of the start position x_(s) and end position x_(e) of anintegral range is determined with the center pixel position (x, y) andpixel width of an output pixel to be generated from now, and the shiftamount C_(x)(y) representing the direction of data continuity. Note that(x, y) represents the relative position from the center position of apixel of interest when the actual world estimating unit 102 generatesthe approximation function f(x).

Further, each of the center pixel position (x, y) and pixel width of anoutput pixel to be generated from now is determined with the spatialresolution power (integral range) set by the conditions setting unit3121.

Also, with the shift amount C_(x)(y), and the angle θ supplied from thedata continuity detecting unit 101, the relation such as the followingExpression (112) and Expression (113) holds, and accordingly, the shiftamount C_(x)(y) is determined with the angle θ.

$\begin{matrix}{G_{f} = {{\tan\;\theta} = \frac{\mathbb{d}y}{\mathbb{d}x}}} & (112) \\{{C_{x}(y)} = \frac{y}{G_{f}}} & (113)\end{matrix}$

Note that in Expression (112), G_(f) represents a gradient representingthe direction of data continuity, θ represents an angle (angle generatedbetween the X direction serving as one direction of the spatialdirections and the direction of data continuity represented with agradient G_(f)) of one of the data continuity information output fromthe data continuity detecting unit 101 (FIG. 89). Also, dx representsthe amount of fine movement in the X direction, and dy represents theamount of fine movement in the Y direction (spatial directionperpendicular to the X direction) as to the dx.

Accordingly, the integral component calculation unit 3123 calculates theintegral components K_(i)(x_(s), x_(e)) based on the number ofdimensions and spatial resolution power (integral range) set by theconditions setting unit 3121, and the angle θ of the data continuityinformation output from the data continuity detecting unit 101, andsupplies the calculated results to the output pixel value calculationunit 3124 as an integral component table.

The output pixel value calculation unit 3124 calculates the right sideof the above Expression (110) using the features table supplied from thefeatures storage unit 3122 and the integral component table suppliedfrom the integral component calculation unit 3123, and outputs thecalculation result as an output pixel value M.

Next, description will be made regarding image generating processing(processing in step S103 in FIG. 29) by the image generating unit 103(FIG. 92) employing the one-dimensional reintegration method withreference to the flowchart in FIG. 93.

For example, now, let us say that the actual world estimating unit 102has already generated the approximation function f(x) such as shown inFIG. 90 while taking the pixel 3101 such as shown in FIG. 90 describedabove as a pixel of interest at the processing in step S102 in FIG. 29described above.

Also, let us say that the data continuity detecting unit 101 has alreadyoutput the angle θ such as shown in FIG. 90 as data continuityinformation at the processing in step S101 in FIG. 29 described above.

In this case, the conditions setting unit 3121 sets conditions (thenumber of dimensions and an integral range) at step S3101 in FIG. 93.

For example, now, let us say that 5 has been set as the number ofdimensions, and also a spatial quadruple density (spatial resolutionpower to cause the pitch width of a pixel to become half power in theupper/lower/left/right sides) has been set as an integral range.

That is to say, in this case, consequently, it has been set that thefour pixel 3111 through pixel 3114 are created newly in a range of −0.5through 0.5 in the X direction, and also a range of −0.5 through 0.5 inthe Y direction (in the range of the pixel 3101 in FIG. 90), such asshown in FIG. 91.

In step S3102, the features storage unit 3122 acquires the features ofthe approximation function f(x) supplied from the actual worldestimating unit 102, and generates a features table. In this case,coefficients w₀ through W₅ of the approximation function f(x) serving asa five-dimensional polynomial are supplied from the actual worldestimating unit 102, and accordingly, (w₀, w₁, w₂, w₃, w₄, w₅) isgenerated as a features table.

In step S3103, the integral component calculation unit 3123 calculatesintegral components based on the conditions (the number of dimensionsand integral range) set by the conditions setting unit 3121, and thedata continuity information (angle θ) supplied from the data continuitydetecting unit 101, and generates an integral component table.

Specifically, for example, if we say that the respective pixels 3111through 3114, which are to be generated from now, are appended withnumbers (hereafter, such a number is referred to as a mode number) 1through 4, the integral component calculation unit 3123 calculates theintegral components K_(i)(x_(s), x_(e)) of the above Expression (111) asa function of l (however, l represents a mode number) such as integralcomponents K_(i)(l) shown in the left side of the following Expression(114).K _(i)(l)=K _(i)(x _(s) , x _(e))  (114)

Specifically, in this case, the integral components K_(i)(l) shown inthe following Expression (115) are calculated.

$\begin{matrix}\begin{matrix}{{k_{i}(1)} = {k_{i}\left( {{{- 0.5} - {C_{x}\left( {- 0.25} \right)}},{0 - {C_{x}\left( {- 0.25} \right)}}} \right)}} \\{{k_{i}(2)} = {k_{i}\left( {{0 - {C_{x}\left( {- 0.25} \right)}},{0.5 - {C_{x}\left( {- 0.25} \right)}}} \right)}} \\{{k_{i}(3)} = {k_{i}\left( {{{- 0.5} - {C_{x}(0.25)}},{0 - {C_{x}(0.25)}}} \right)}} \\{{k_{i}(4)} = {k_{i}\left( {{0 - {C_{x}(0.25)}},{0.5 - {C_{x}(0.25)}}} \right)}}\end{matrix} & (115)\end{matrix}$

Note that in Expression (115), the left side represents the integralcomponents K_(i)(l), and the right side represents the integralcomponents K_(i)(x_(s), x_(e)). That is to say, in this case, l is anyone of 1 through 4, and also i is any one of 0 through 5, andaccordingly, 24 K_(i)(l) in total of 6 K_(i)(1), 6 K_(i)(2), 6 K_(i)(3),and 6 K_(i)(4) are calculated.

More specifically, first, the integral component calculation unit 3123calculates each of the shift amounts C_(x)(−0.25) and C_(x)(0.25) fromthe above Expression (112) and Expression (113) using the angle θsupplied from the data continuity detecting unit 101.

Next, the integral component calculation unit 3123 calculates theintegral components K_(i)(x_(s), x_(e)) of each right side of the fourexpressions in Expression (115) regarding i=0 through 5 using thecalculated shift amounts C_(x)(−0.25) and C_(x)(0.25). Note that withthis calculation of the integral components K_(i)(x_(s), x_(e)), theabove Expression (111) is employed.

Subsequently, the integral component calculation unit 3123 converts eachof the 24 integral components K_(i)(x_(s), x_(e)) calculated into thecorresponding integral components K_(i)(l) in accordance with Expression(115), and generates an integral component table including the 24integral components K_(i)(l) converted (i.e., 6 K_(i)(1), 6 K_(i)(2), 6K_(i)(3), and 6 K_(i)(4)).

Note that the sequence of the processing in step S3102 and theprocessing in step S3103 is not restricted to the example in FIG. 93,the processing in step S3103 may be executed first, or the processing instep S3102 and the processing in step S3103 may be executedsimultaneously.

Next, in step S3104, the output pixel value calculation unit 3124calculates the output pixel values M(1) through M(4) respectively basedon the features table generated by the features storage unit 3122 at theprocessing in step S3102, and the integral component table generated bythe integral component calculation unit 3123 at the processing in stepS3103.

Specifically, in this case, the output pixel value calculation unit 3124calculates each of the pixel value M(1) of the pixel 3111 (pixel of modenumber 1), the pixel value M(2) of the pixel 3112 (pixel of mode number2), the pixel value M(3) of the pixel 3113 (pixel of mode number 3), andthe pixel value M(4) of the pixel 3114 (pixel of mode number 4) bycalculating the right sides of the following Expression (116) throughExpression (119) corresponding to the above Expression (110).

$\begin{matrix}{{M(1)} = {\sum\limits_{i = 0}^{5}{w_{i}{k_{i}(1)}}}} & (116) \\{{M(2)} = {\sum\limits_{i = 0}^{5}{w_{i}{k_{i}(2)}}}} & (117) \\{{M(3)} = {\sum\limits_{i = 0}^{5}{w_{i}{k_{i}(3)}}}} & (118) \\{{M(4)} = {\sum\limits_{i = 0}^{5}{w_{i}{k_{i}(4)}}}} & (119)\end{matrix}$

In step S3105, the output pixel value calculation unit 3124 determinesregarding whether or not the processing of all the pixels has beencompleted.

In step S3105, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S3102, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S3102 through S3104 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S3105, in the event that determination is made that theprocessing of all the pixels has been completed), the output pixel valuecalculation unit 3124 outputs the image in step S3106. Then, the imagegenerating processing ends.

Next, description will be made regarding the differences between theoutput image obtained by employing the one-dimensional reintegrationmethod and the output image obtained by employing another method(conventional classification adaptation processing) regarding apredetermined input image with reference to FIG. 94 through FIG. 101.

FIG. 94 is a diagram illustrating the original image of the input image,and FIG. 95 illustrates image data corresponding to the original imagein FIG. 94. In FIG. 95, the axis in the vertical direction in thedrawing represents pixel values, and the axis in the lower rightdirection in the drawing represents the X direction serving as onedirection of the spatial directions of the image, and the axis in theupper right direction in the drawing represents the Y direction servingas the other direction of the spatial directions of the image. Note thatthe respective axes in later-described FIG. 97, FIG. 99, and FIG. 101corresponds to the axes in FIG. 95.

FIG. 96 is a diagram illustrating an example of an input image. Theinput image illustrated in FIG. 96 is an image generated by taking themean of the pixel values of the pixels belonged to a block made up of2×2 pixels shown in FIG. 94 as the pixel value of one pixel. That is tosay, the input image is an image obtained by integrating the image shownin FIG. 94 in the spatial direction, which imitates the integrationproperty of a sensor. Also, FIG. 97 illustrates image data correspondingto the input image in FIG. 96.

The original image illustrated in FIG. 94 includes a fine-line imageinclined almost 5° clockwise from the vertical direction. Similarly, theinput image illustrated in FIG. 96 includes a fine-line image inclinedalmost 5° clockwise from the vertical direction.

FIG. 98 is a diagram illustrating an image (hereafter, the imageillustrated in FIG. 98 is referred to as the original image) obtained bysubjecting the input image illustrated in FIG. 96 to conventional classclassification adaptation processing. Also, FIG. 99 illustrates imagedata corresponding to the conventional image.

Note that the class classification adaptation processing is made up ofclassification processing and adaptation processing, data is classifiedbased on the property thereof by the class classification processing,and is subjected to the adaptation processing for each class, asdescribed above. With the adaptation processing, for example, alow-quality or standard-quality image is subjected to mapping using apredetermined tap coefficient so as to be converted into a high-qualityimage.

FIG. 100 is a diagram illustrating an image (hereafter, the imageillustrated in FIG. 100 is referred to as a reintegration image)obtained by applying the one-dimensional reintegration method, to theinput image illustrated in FIG. 96. Also, FIG. 101 illustrates imagedata corresponding to the reintegration image.

It can be understood that upon the conventional image in FIG. 98 beingcompared with the reintegration image in FIG. 100, a fine-line image isdifferent from that in the original image in FIG. 94 in the conventionalimage, but on the other hand, the fine-line image is almost the same asthat in the original image in FIG. 94 in the reintegration image.

This difference is caused by a difference wherein the conventional classclassification adaptation processing is a method for performingprocessing on the basis (origin) of the input image in FIG. 96, but onthe other hand, the one-dimensional reintegration method is a method forestimating the original image in FIG. 94 (generating the approximationfunction f(x) corresponding to the original image) in light ofcontinuity of a fine line, and performing processing (performingreintegration so as to calculate pixel values) on the basis (origin) ofthe original image estimated.

Thus, with the one-dimensional reintegration method, the output image(pixel value) is generated by integrating the approximation functionf(x) in an arbitrary range, using an approximation function f(x) such asa primary polynomial generated by a one-dimensional approximation method(the approximation function f(x) of the X-cross section waveform F(x) ofthe actual world) as a standard (origin).

Accordingly, with the one-dimensional reintegration method, it becomespossible to output an image more similar to the original image (thelight signal in the actual world 1 which is to be cast in the sensor 2)in comparison with the conventional other methods.

Also, with the one-dimensional reintegration method, as described above,the integral range is arbitrary, and accordingly, it becomes possible tocreate resolution (temporal resolution or spatial resolution) differentfrom the resolution of an input image by varying the integral range.That is to say, it becomes possible to generate an image havingarbitrary powered resolution as well as an integer value as to theresolution of the input image.

Further, the one-dimensional reintegration method enables calculation ofan output image (pixel values) with less calculation processing amountthan other reintegration methods.

Next, description will be made regarding a two-dimensional reintegrationmethod with reference to FIG. 102 through FIG. 108.

The two-dimensional reintegration method is based on condition that theapproximation function f(x, y) has been generated with thetwo-dimensional polynomial approximating method.

That is to say, for example, it is an assumption that the image functionF(x, y, t) representing the light signal in the actual world 1 (FIG. 89)having continuity in the spatial direction represented with the gradientG_(F) has been approximated with a waveform projected in the spatialdirections (X direction and Y direction), i.e., the waveform F(x, y) onthe X-Y plane has been approximated with the approximation function f(x,y) serving as a n-dimensional (n is an arbitrary integer) polynomial,such as shown in FIG. 102.

In FIG. 102, the horizontal direction represents the X direction servingas one direction in the spatial directions, the upper right directionrepresents the Y direction serving as the other direction in the spatialdirections, and the vertical direction represents light levels,respectively in the drawing. G_(F) represents gradient as continuity inthe spatial directions.

Note that with the example in FIG. 102, the direction of continuity istaken as the spatial directions (X direction and Y direction), so theprojection function of a light signal to be approximated is taken as thefunction F(x, y), but as described later, the function F(x, t) orfunction F(y, t) may be a target of approximation according to thedirection of continuity.

In the case of the example in FIG. 102, with the two-dimensionalreintegration method, the output pixel value M is calculated as thefollowing Expression (120).

$\begin{matrix}{M = {G_{e} \times {\int_{y_{s}}^{y_{e}}{\int_{x_{s}}^{x_{e}}{{f\left( {x,y} \right)}{\mathbb{d}x}{\mathbb{d}y}}}}}} & (120)\end{matrix}$

Note that in Expression (120), y_(s) represents an integration startposition in the Y direction, and y_(e) represents an integration endposition in the Y direction. Similarly, x_(s) represents an integrationstart position in the X direction, and x_(e) represents an integrationend position in the X direction. Also, G_(e) represents a predeterminedgain.

In Expression (120), an integral range can be set arbitrarily, andaccordingly, with the two-dimensional reintegration method, it becomespossible to create pixels having an arbitrary powered spatial resolutionas to the original pixels (the pixels of an input image from the sensor2 (FIG. 89)) without deterioration by appropriately changing thisintegral range.

FIG. 103 represents a configuration example of the image generating unit103 employing the two-dimensional reintegration method.

As shown in FIG. 103, the image generating unit 103 in this exampleincludes a conditions setting unit 3201, features storage unit 3202,integral component calculation unit 3203, and output pixel valuecalculation unit 3204.

The conditions setting unit 3201 sets the number of dimensions n of theapproximation function f(x, y) based on the actual world estimatinginformation (with the example in FIG. 103, the features of theapproximation function f(x, y)) supplied from the actual worldestimating unit 102.

The conditions setting unit 3201 also sets an integral range in the caseof reintegrating the approximation function f(x, y) (in the case ofcalculating an output pixel value). Note that an integral range set bythe conditions setting unit 3201 does not need to be the vertical widthor the horizontal width of a pixel. For example, the approximationfunction f(x, y) is integrated in the spatial directions (X directionand Y direction), and accordingly, a specific integral range can bedetermined as long as the relative size (power of spatial resolution) ofan output pixel (pixel to be generated from now by the image generatingunit 103) as to the spatial size of each pixel of an input image fromthe sensor 2 is known. Accordingly, the conditions setting unit 3201 canset, for example, a spatial resolution power as an integral range.

The features storage unit 3202 temporally stores the features of theapproximation function f(x, y) sequentially supplied from the actualworld estimating unit 102. Subsequently, upon the features storage unit3202 storing all of the features of the approximation function f(x, y),the features storage unit 3202 generates a features table including allof the features of the approximation function f(x, y), and supplies thisto the output pixel value calculation unit 3204.

Now, description will be made regarding the details of the approximationfunction f(x, y).

For example, now, let us say that the light signals (light signalsrepresented with the wave F(x, y)) in the actual world 1 (FIG. 89)having continuity in the spatial directions represented with thegradient G_(F) shown in FIG. 102 described above have been detected bythe sensor 2 (FIG. 89), and have been output as an input image (pixelvalues).

Further, for example, let us say that the data continuity detecting unit101 (FIG. 3) has subjected a region 3221 of an input image made up of 20pixels in total (20 squares represented with a dashed line in thedrawing) of 4 pixels in the X direction and also 5 pixels in the Ydirection of this input image to the processing thereof, and has outputan angle θ (angle θ generated between the direction of data continuityrepresented with the gradient G_(f) corresponding to the gradient G_(F)and the X direction) as one of data continuity information, as shown inFIG. 104.

Note that as viewed from the actual world estimating unit 102, the datacontinuity detecting unit 101 should simply output the angle θ at apixel of interest, and accordingly, the processing region of the datacontinuity detecting unit 101 is not restricted to the above region 3221in the input image.

Also, with the region 3221 in the input image, the horizontal directionin the drawing represents the X direction serving as one direction ofthe spatial directions, and the vertical direction in the drawingrepresents the Y direction serving the other direction of the spatialdirections.

Further, in FIG. 104, a pixel, which is the second pixel from the left,and also the third pixel from the bottom, is taken as a pixel ofinterest, and an (x, y) coordinates system is set so as to take thecenter of the pixel of interest as the origin (0, 0). A relativedistance (hereafter, referred to as a cross-sectional directiondistance) in the X direction as to a straight line (straight line of thegradient G_(f) representing the direction of data continuity) having anangle θ passing through the origin (0, 0) is taken as x′.

Further, in FIG. 104, the graph on the right side represents theapproximation function f(x′) serving as a n-dimensional (n is anarbitrary integer) polynomial, which is a function approximating aone-dimensional waveform (hereafter, referred to as an X cross-sectionalwaveform F(x′)) wherein the image function F(x, y, t) of which variablesare positions x, y, and z on the three-dimensional space, andpoint-in-time t is projected in the X direction at an arbitrary positiony in the Y direction. Of the axes in the graph on the right side, theaxis in the horizontal direction in the drawing represents across-sectional direction distance, and the axis in the verticaldirection in the drawing represents pixel values.

In this case, the approximation function f(X′) shown in FIG. 104 is an-dimensional polynomial, so is represented as the following Expression(121).

$\begin{matrix}{{f\left( x^{\prime} \right)} = {{w_{0} + {w_{1}x^{\prime}} + {w_{2}x^{\prime}} + \ldots + {w_{n}x^{\prime\; n}}} = {\sum\limits_{i = 0}^{n}{w_{i}x^{\prime\; i}}}}} & (121)\end{matrix}$

Also, since the angle θ is determined, the straight line having angle θpassing through the origin (0, 0) is uniquely determined, and a positionx₁ in the X direction of the straight line at an arbitrary position y inthe Y direction is represented as the following Expression (122).However, in Expression (122), s represents cot θ.x ₁ =s×y  (122)

That is to say, as shown in FIG. 104, a point on the straight linecorresponding to continuity of data represented with the gradient G_(f)is represented with a coordinate value (x₁, y).

The cross-sectional direction distance X′ is represented as thefollowing Expression (123) using Expression (122).x′=x−x ₁ =x−s×y  (123)

Accordingly, the approximation function f(x, y) at an arbitrary position(x, y) within the input image region 3221 is represented as thefollowing Expression (124) using Expression (121) and Expression (123).

$\begin{matrix}{{f\left( {x,y} \right)} = {\sum\limits_{i = 0}^{n}{w_{i}\left( {x - {s \times y}} \right)}}} & (124)\end{matrix}$

Note that in Expression (124), w_(i) represents the features of theapproximation function f(x, y).

Now, description will return to FIG. 103, wherein the features w_(i)included in Expression (124) are supplied from the actual worldestimating unit 102, and stored in the features storage unit 3202. Uponthe features storage unit 3202 storing all of the features w_(i)represented with Expression (124), the features storage unit 3202generates a features table including all of the features w_(i), andsupplies this to the output pixel value calculation unit 3204.

Also, upon the right side of the above Expression (120) being expanded(calculated) by substituting the approximation function f(x, y) ofExpression (124) for the approximation function f(x, y) in the rightside of Expression (120), the output pixel value M is represented as thefollowing Expression (125).

$\begin{matrix}\begin{matrix}{M = {G_{e} \times {\sum\limits_{i = 0}^{n}{w_{i} \times \frac{\begin{Bmatrix}{\left( {x_{e} - {s\; x\; y_{e}}} \right)^{i + 2} - \left( {x_{e} - {s\; x\; y_{s}}} \right)^{i + 2} -} \\{\left( {x_{s} - {s\; x\; y_{e}}} \right)^{i + 2} + \left( {x_{s} - {s\; x\; y_{s}}} \right)^{i + 2}}\end{Bmatrix}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)}}}}} \\{= {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}\left( {x_{s},x_{e},y_{s},y_{e}} \right)}}}}\end{matrix} & (125)\end{matrix}$

In Expression (125), K_(i)(x_(s), x_(e), y_(s), y_(e)) represent theintegral components of the i-dimensional term. That is to say, theintegral components K_(i)(x_(s), x_(e), y_(s), y_(e)) are such as shownin the following Expression (126).

$\begin{matrix}{{k_{i}\left( {x_{s},x_{e},y_{s},y_{e}} \right)} = {G_{e} \times \frac{\begin{Bmatrix}{\left( {x_{e} - {s\; x\; y_{e}}} \right)^{i + 2} - \left( {x_{e} - {s\; x\; y_{s}}} \right)^{i + 2} -} \\{\left( {x_{s} - {s\; x\; y_{e}}} \right)^{i + 2} + \left( {x_{s} - {s\; x\; y_{s}}} \right)^{i + 2}}\end{Bmatrix}}{{s\left( {i + 1} \right)}\left( {i + 2} \right)}}} & (126)\end{matrix}$

The integral component calculation unit 3203 calculates the integralcomponents K_(i)(x_(s), x_(e), y_(s), y_(e)). Specifically, as shown inExpression (125) and Expression (126), the integral componentsK_(i)(x_(s), x_(e), y_(s), y_(e)) can be calculated as long as the startposition x_(s) in the X direction and end position x_(e) in the Xdirection of an integral range, the start position y_(s) in the Ydirection and end position y_(e) in the Y direction of an integralrange, variable s, gain G_(e), and i of the i-dimensional term areknown.

Of these, the gain G_(e) is determined with the spatial resolution power(integral range) set by the conditions setting unit 3201.

The range of i is determined with the number of dimensions n set by theconditions setting unit 3201.

A variable s is, as described above, cot θ, so is determined with theangle θ output from the data continuity detecting unit 101.

Also, each of the start position x_(s) in the X direction and endposition x_(e) in the X direction of an integral range, and the startposition y_(s) in the Y direction and end position y_(e) in the Ydirection of an integral range is determined with the center pixelposition (x, y) and pixel width of an output pixel to be generated fromnow. Note that (x, y) represents a relative position from the centerposition of the pixel of interest when the actual world estimating unit102 generates the approximation function f(x).

Further, each of the center pixel position (x, y) and pixel width of anoutput pixel to be generated from now is determined with the spatialresolution power (integral range) set by the conditions setting unit3201.

Accordingly, the integral component calculation unit 3203 calculatesintegral components K_(i)(x_(s), x_(e), y_(s), y_(e)) based on thenumber of dimensions and the spatial resolution power (integral range)set by the conditions setting unit 3201, and the angle θ of the datacontinuity information output from the data continuity detecting unit101, and supplies the calculated result to the output pixel valuecalculation unit 3204 as an integral component table.

The output pixel value calculation unit 3204 calculates the right sideof the above Expression (125) using the features table supplied from thefeatures storage unit 3202, and the integral component table suppliedfrom the integral component calculation unit 3203, and outputs thecalculated result to the outside as the output pixel value M.

Next, description will be made regarding image generating processing(processing in step S103 in FIG. 29) by the image generating unit 103(FIG. 104) employing the two-dimensional reintegration method withreference to the flowchart in FIG. 105.

For example, let us say that the light signals represented with thefunction F(x, y) shown in FIG. 102 have been cast in the sensor 2 so asto become an input image, and the actual world estimating unit 102 hasalready generated the approximation function f(x, y) for approximatingthe function F(x, y) with one pixel 3231 such as shown in FIG. 106 as apixel of interest at the processing in step S102 in FIG. 29 describedabove.

Note that in FIG. 106, the pixel value (input pixel value) of the pixel3231 is taken as P, and the shape of the pixel 3231 is taken as a squareof which one side is 1 in length. Also, of the spatial directions, thedirection in parallel with one side of the pixel 3231 is taken as the Xdirection, and the direction orthogonal to the X direction is taken asthe Y direction. Further, a coordinates system (hereafter, referred toas a pixel-of-interest coordinates system) in the spatial directions (Xdirection and Y direction) of which the origin is the center of thepixel 3231 is set.

Also, let us say that in FIG. 106, the data continuity detecting unit101, which takes the pixel 3231 as a pixel of interest, has alreadyoutput the angle θ as data continuity information corresponding tocontinuity of data represented with the gradient G_(f) at the processingin step S101 in FIG. 29 described above.

Description will return to FIG. 105, and in this case, the conditionssetting unit 3201 sets conditions (the number of dimensions and anintegral range) at step S3201.

For example, now, let us say that 5 has been set as the number ofdimensions, and also spatial quadruple density (spatial resolution powerto cause the pitch width of a pixel to become half power in theupper/lower/left/right sides) has been set as an integral range.

That is to say, in this case, it has been set that the four pixel 3241through pixel 3244 are created newly in a range of −0.5 through 0.5 inthe X direction, and also a range of −0.5 through 0.5 in the Y direction(in the range of the pixel 3231 in FIG. 106), such as shown in FIG. 107.Note that in FIG. 107 as well, the same pixel-of-interest coordinatessystem as that in FIG. 106 is shown.

Also, in FIG. 107, M(1) represents the pixel value of the pixel 3241 tobe generated from now, M(2) represents the pixel value of the pixel 3242to be generated from now, M(3) represents the pixel value of the pixel3243 to be generated from now, and M(4) represents the pixel value ofthe pixel 3244 to be generated from now.

Description will return to FIG. 105, in step S3202, the features storageunit 3202 acquires the features of the approximation function f(x, y)supplied from the actual world estimating unit 102, and generates afeatures table. In this case, the coefficients w₀ through W₅ of theapproximation function f(x) serving as a 5-dimensional polynomial aresupplied from the actual world estimating unit 102, and accordingly,(w₀, w₁, w₂, w₃, w₄, w₅) is generated as a features table.

In step S3203, the integral component calculation unit 3203 calculatesintegral components based on the conditions (the number of dimensionsand an integral range) set by the conditions setting unit 3201, and thedata continuity information (angle θ) supplied from the data continuitydetecting unit 101, and generates an integral component table.

Specifically, for example, let us say that numbers (hereafter, such anumber is referred to as a mode number) 1 through 4 are respectivelyappended to the pixel 3241 through pixel 3244 to be generated from now,the integral component calculation unit 3203 calculates the integralcomponents K_(i)(x_(s), x_(e), y_(s), y_(e)) of the above Expression(125) as a function of l (however, l represents a mode number) such asthe integral components K_(i)(l) shown in the left side of the followingExpression (127).K _(i)(l)=K _(i)(x _(s) , x _(e) , y _(s) , y _(e))  (127)

Specifically, in this case, the integral components K_(i)(l) shown inthe following Expression (128) are calculated.

$\begin{matrix}\begin{matrix}{{k_{i}(1)} = {k_{i}\left( {{- 0.5},0,0,0.5} \right)}} \\{{k_{i}(2)} = {k_{i}\left( {0,0.5,0,0.5} \right)}} \\{{k_{i}(3)} = {k_{i}\left( {{- 0.5},0,{- 0.5},0} \right)}} \\{{k_{i}(4)} = {k_{i}\left( {0,0.5,{- 0.5},0} \right)}}\end{matrix} & (128)\end{matrix}$

Note that in Expression (128), the left side represents the integralcomponents K_(i)(l), and the right side represents the integralcomponents K_(i)(x_(s), x_(e), y_(s), y_(e)). That is to say, in thiscase, l is any one of 1 thorough 4, and also i is any one of 0 through5, and accordingly, 24 K_(i)(l) in total of 6 K_(i)(1), 6 K_(i)(2), 6K_(i)(3), and 6 K_(i)(4) are calculated.

More specifically, first, the integral component calculation unit 3203calculates the variable s (s=cot θ) of the above Expression (122) usingthe angle θ supplied from the data continuity detecting unit 101.

Next, the integral component calculation unit 3203 calculates theintegral components K_(i)(x_(s), x_(e), y_(s), y_(e)) of each right sideof the four expressions in Expression (128) regarding i=0 through 5using the calculated variable s. Note that with this calculation of theintegral components K_(i)(x_(s), x_(e), y_(s), y_(e)), the aboveExpression (125) is employed.

Subsequently, the integral component calculation unit 3203 converts eachof the 24 integral components K_(i)(x_(s), x_(e), y_(s), y_(e))calculated into the corresponding integral components K_(i)(l) inaccordance with Expression (128), and generates an integral componenttable including the 24 integral components K_(i)(l) converted (i.e., 6K_(i)(1), 6 K_(i)(2), 6 K_(i)(3), and 6 K_(i)(4)).

Note that the sequence of the processing in step S3202 and theprocessing in step S3203 is not restricted to the example in FIG. 105,the processing in step S3203 may be executed first, or the processing instep S3202 and the processing in step S3203 may be executedsimultaneously.

Next, in step S3204, the output pixel value calculation unit 3204calculates the output pixel values M(1) through M(4) respectively basedon the features table generated by the features storage unit 3202 at theprocessing in step S3202, and the integral component table generated bythe integral component calculation unit 3203 at the processing in stepS3203.

Specifically, in this case, the output pixel value calculation unit 3204calculates each of the pixel value M(1) of the pixel 3241 (pixel of modenumber 1), the pixel value M(2) of the pixel 3242 (pixel of mode number2), the pixel value M(3) of the pixel 3243 (pixel of mode number 3), andthe pixel value M(4) of the pixel 3244 (pixel of mode number 4) shown inFIG. 107 by calculating the right sides of the following Expression(129) through Expression (132) corresponding to the above Expression(125).

$\begin{matrix}{{M(1)} = {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}(1)}}}} & (129) \\{{M(2)} = {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}(2)}}}} & (130) \\{{M(3)} = {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}(3)}}}} & (131) \\{{M(4)} = {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}(4)}}}} & (132)\end{matrix}$

However, in this case, each n of Expression (129) through Expression(132) becomes 5.

In step S3205, the output pixel value calculation unit 3204 determinesregarding whether or not the processing of all the pixels has beencompleted.

In step S3205, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S3202, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S3202 through S3204 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S3205, in the event that determination is made that theprocessing of all the pixels has been completed), the output pixel valuecalculation unit 3204 outputs the image in step S3206. Then, the imagegenerating processing ends.

Thus, four pixels having higher spatial resolution than the input pixel3231, i.e., the pixel 3241 through pixel 3244 (FIG. 107) can be createdby employing the two-dimensional reintegration method as a pixel at thepixel 3231 of the input image (FIG. 106) from the sensor 2 (FIG. 89).Further, though not shown in the drawing, as described above, the imagegenerating unit 103 can create a pixel having an arbitrary poweredspatial resolution as to the input pixel 3231 without deterioration byappropriately changing an integral range, in addition to the pixel 3241through pixel 3244.

As described above, as description of the two-dimensional reintegrationmethod, an example for subjecting the approximation function f(x, y) asto the spatial directions (X direction and Y direction) totwo-dimensional integration has been employed, but the two-dimensionalreintegration method can be applied to the time-space directions (Xdirection and t direction, or Y direction and t direction).

That is to say, the above example is an example in the case in which thelight signals in the actual world 1 (FIG. 89) have continuity in thespatial directions represented with the gradient G_(F) such as shown inFIG. 102, and accordingly, an expression including two-dimensionalintegration in the spatial directions (X direction and Y direction) suchas shown in the above Expression (120) has been employed. However, theconcept regarding two-dimensional integration can be applied not only tothe spatial direction but also the time-space directions (X directionand t direction, or Y direction and t direction).

In other words, with the two-dimensional polynomial approximating methodserving as an assumption of the two-dimensional reintegration method, itis possible to perform approximation using a two-dimensionalapproximation function f even in the case in which the image functionF(x, y, t) representing the light signals has continuity in thetime-space directions (however, X direction and t direction, or Ydirection and t direction) as well as continuity in the spatialdirections.

Specifically, for example, in the event that there is an object movinghorizontally in the X direction at uniform velocity, the direction ofmovement of the object is represented with like a gradient V_(F) in theX-t plane such as shown in FIG. 108. In other words, it can be said thatthe gradient V_(F) represents the direction of continuity in thetime-space directions in the X-t plane. Accordingly, the data continuitydetecting unit 101 (FIG. 89) can output movement θ such as shown in FIG.108 (strictly speaking, though not shown in the drawing, movement θ isan angle generated by the direction of data continuity represented withthe gradient V_(f) corresponding to the gradient V_(F) and the Xdirection in the spatial direction) as data continuity informationcorresponding to the gradient V_(F) representing continuity in thetime-space directions in the X-t plane as well as the above angle θ(data continuity information corresponding to the gradient G_(F)representing continuity in the spatial directions in the X-Y plane).

Also, the actual world estimating unit 102 (FIG. 89) employing thetwo-dimensional polynomial approximating method can calculate thecoefficients (features) w_(i) of an approximation function f(x, t) withthe same method as the above method by employing the movement θ insteadof the angle θ described above. However, in this case, the equation tobe employed is not the above Expression (124) but the followingExpression (133).

$\begin{matrix}{{f\left( {x,y} \right)} = {\sum\limits_{i = 0}^{n}{w_{i}\left( {x - {s \times t}} \right)}}} & (133)\end{matrix}$

Note that in Expression (133), s is cot θ (however, θ is movement).

Accordingly, the image generating unit 103 (FIG. 89) employing thetwo-dimensional reintegration method can calculate the pixel value M bysubstituting the f(x, t) of the above Expression (133) for the rightside of the following Expression (134), and calculating this.

$\begin{matrix}{M = {G_{e} \times {\int_{t_{s}}^{t_{e}}{\int_{x_{s}}^{x_{e}}{{f\left( {x,t} \right)}{\mathbb{d}x}{\mathbb{d}t}}}}}} & (134)\end{matrix}$

Note that in Expression (134), t_(s), represents an integration startposition in the t direction, and t_(e) represents an integration endposition in the t direction. Similarly, x_(s) represents an integrationstart position in the X direction, and x_(e) represents an integrationend position in the X direction. G_(e) represents a predetermined gain.

Alternately, an approximation function f(y, t) focusing attention on thespatial direction Y instead of the spatial direction X can be handled asthe same way as the above approximation function f(x, t).

Incidentally, in Expression (133), it becomes possible to obtain datanot integrated in the temporal direction, i.e., data without movementblurring by regarding the t direction as constant, i.e., by performingintegration while ignoring integration in the t direction. In otherwords, this method may be regarded as one of two-dimensionalreintegration methods in that reintegration is performed on conditionthat one certain dimension of the two-dimensional approximation functionf is constant, or in fact, may be regarded as one of one-dimensionalreintegration methods in that one-dimensional reintegration in the Xdirection is performed.

Also, in Expression (134), an integral range may be set arbitrarily, andaccordingly, with the two-dimensional reintegration method, it becomespossible to create a pixel having an arbitrary powered resolution as tothe original pixel (pixel of an input image from the sensor 2 (FIG. 89))without deterioration by appropriately changing this integral range.

That is to say, with the two-dimensional reintegration method, itbecomes possible to create temporal resolution by appropriately changingan integral range in the temporal direction t. Also, it becomes possibleto create spatial resolution by appropriately changing an integral rangein the spatial direction X (or spatial direction Y). Further, it becomespossible to create both temporal resolution and spatial resolution byappropriately changing each integral range in the temporal direction tand in the spatial direction X.

Note that as described above, creation of any one of temporal resolutionand spatial resolution may be performed even with the one-dimensionalreintegration method, but creation of both temporal resolution andspatial resolution cannot be performed with the one-dimensionalreintegration method in theory, which becomes possible only byperforming two-dimensional or more reintegration. That is to say,creation of both temporal resolution and spatial resolution becomespossible only by employing the two-dimensional reintegration method anda later-described three-dimensional reintegration method.

Also, the two-dimensional reintegration method takes not one-dimensionalbut two-dimensional integration effects into consideration, andaccordingly, an image more similar to the light signal in the actualworld 1 (FIG. 89) may be created.

Next, description will be made regarding a three-dimensionalreintegration method with reference to FIG. 109 and FIG. 110.

With the three-dimensional reintegration method, the approximationfunction f(x, y, t) has been created using the three-dimensionalapproximating method, which is an assumption.

In this case, with the three-dimensional reintegration method, theoutput pixel value M is calculated as the following Expression (135).

$\begin{matrix}{M = {G_{e} \times {\int_{t_{s}}^{t_{e}}{\int_{y_{s}}^{y_{e}}{\int_{x_{s}}^{x_{e}}{{f\left( {x,y,t} \right)}{\mathbb{d}x}{\mathbb{d}y}{\mathbb{d}t}}}}}}} & (135)\end{matrix}$

Note that in Expression (135), t_(s) represents an integration startposition in the t direction, and t_(e) represents an integration endposition in the t direction. Similarly, y_(s) represents an integrationstart position in the Y direction, and y_(e) represents an integrationend position in the Y direction. Also, x_(s) represents an integrationstart position in the X direction, and x_(e) represents an integrationend position in the X direction. G_(e) represents a predetermined gain.

Also, in Expression (135), an integral range may be set arbitrarily, andaccordingly, with the three-dimensional reintegration method, it becomespossible to create a pixel having an arbitrary powered time-spaceresolution as to the original pixel (pixel of an input image from thesensor 2 (FIG. 89)) without deterioration by appropriately changing thisintegral range. That is to say, upon the integral range in the spatialdirection being reduced, a pixel pitch can be reduced without restraint.On the other hand, upon the integral range in the spatial directionbeing enlarged, a pixel pitch can be enlarged without restraint. Also,upon the integral range in the temporal direction being reduced,temporal resolution can be created based on an actual waveform.

FIG. 109 represents a configuration example of the image generating unit103 employing the three-dimensional reintegration method.

As shown in FIG. 109, this example of the image generating unit 103includes a conditions setting unit 3301, features storage unit 3302,integral component calculation unit 3303, and output pixel valuecalculation unit 3304.

The conditions setting unit 3301 sets the number of dimensions n of theapproximation function f(x, y, t) based on the actual world estimatinginformation (with the example in FIG. 109, features of the approximationfunction f(x, y, t)) supplied from the actual world estimating unit 102.

The conditions setting unit 3301 sets an integral range in the case ofreintegrating the approximation function f(x, y, t) (in the case ofcalculating output pixel values). Note that an integral range set by theconditions setting unit 3301 needs not to be the width (vertical widthand horizontal width) of a pixel or shutter time itself. For example, itbecomes possible to determine a specific integral range in the spatialdirection as long as the relative size (spatial resolution power) of anoutput pixel (pixel to be generated from now by the image generatingunit 103) as to the spatial size of each pixel of an input image fromthe sensor 2 (FIG. 89) is known. Similarly, it becomes possible todetermine a specific integral range in the temporal direction as long asthe relative time (temporal resolution power) of an output pixel valueas to the shutter time of the sensor 2 (FIG. 89) is known. Accordingly,the conditions setting unit 3301 can set, for example, a spatialresolution power and temporal resolution power as an integral range.

The features storage unit 3302 temporally stores the features of theapproximation function f(x, y, t) sequentially supplied from the actualworld estimating unit 102. Subsequently, upon the features storage unit3302 storing all of the features of the approximation function f(x, y,t), the features storage unit 3302 generates a features table includingall of the features of the approximation function f(x, y, t), andsupplies this to the output pixel value calculation unit 3304.

Incidentally, upon the right side of the approximation function f(x, y)of the right side of the above Expression (135) being expanded(calculated), the output pixel value M is represented as the followingExpression (136).

$\begin{matrix}{M = {\sum\limits_{i = 0}^{n}{w_{i} \times {k_{i}\left( {x_{s},x_{e},y_{s},y_{e},t_{s},t_{e}} \right)}}}} & (136)\end{matrix}$

In Expression (136), K_(i)(x_(s), x_(e), y_(s), y_(e), t_(s), t_(e))represents the integral components of the i-dimensional term. However,x_(s) represents an integration range start position in the X direction,x_(e) represents an integration range end position in the X direction,y_(s) represents an integration range start position in the Y direction,y_(e) represents an integration range end position in the Y direction,t_(s) represents an integration range start position in the t direction,and t_(e) represents an integration range end position in the tdirection, respectively.

The integral component calculation unit 3303 calculates the integralcomponents K_(i)(x_(s), x_(e), y_(s), y_(e), t_(s), t_(e)).Specifically, the integral component calculation unit 3303 calculatesthe integral components K_(i)(x_(s), x_(e), y_(s), y_(e), t_(s), t_(e))based on the number of dimensions and the integral range (spatialresolution power or temporal resolution power) set by the conditionssetting unit 3301, and the angle θ or movement θ of the data continuityinformation output from the data continuity detecting unit 101, andsupplies the calculated results to the output pixel value calculationunit 3304 as an integral component table.

The output pixel value calculation unit 3304 calculates the right sideof the above Expression (136) using the features table supplied from thefeatures storage unit 3302, and the integral component table suppliedfrom the integral component calculation unit 3303, and outputs thecalculated result to the outside as the output pixel value M.

Next, description will be made regarding image generating processing(processing in step S103 in FIG. 29) by the image generating unit 103(FIG. 109) employing the three-dimensional reintegration method withreference to the flowchart in FIG. 110.

For example, let us say that the actual world estimating unit 102 (FIG.89) has already generated an approximation function f(x, y, t) forapproximating the light signals in the actual world 1 (FIG. 89) with apredetermined pixel of an input image as a pixel of interest at theprocessing in step S102 in FIG. 29 described above.

Also, let us say that the data continuity detecting unit 101 (FIG. 89)has already output the angle θ or movement θ as data continuityinformation with the same pixel as the actual world estimating unit 102as a pixel of interest at the processing in step S101 in FIG. 29.

In this case, the conditions setting unit 3301 sets conditions (thenumber of dimensions and an integral range) at step S3301 in FIG. 110.

In step S3302, the features storage unit 3302 acquires the featuresw_(i) of the approximation function f(x, y, t) supplied from the actualworld estimating unit 102, and generates a features table.

In step S3303, the integral component calculation unit 3303 calculatesintegral components based on the conditions (the number of dimensionsand an integral range) set by the conditions setting unit 3301, and thedata continuity information (angle θ or movement θ) supplied from thedata continuity detecting unit 101, and generates an integral componenttable.

Note that the sequence of the processing in step S3302 and theprocessing in step S3303 is not restricted to the example in FIG. 110,the processing in step S3303 may be executed first, or the processing instep S3302 and the processing in step S3303 may be executedsimultaneously.

Next, in step S3304, the output pixel value calculation unit 3304calculates each output pixel value based on the features table generatedby the features storage unit 3302 at the processing in step S3302, andthe integral component table generated by the integral componentcalculation unit 3303 at the processing in step S3303.

In step S3305, the output pixel value calculation unit 3304 determinesregarding whether or not the processing of all the pixels has beencompleted.

In step S3305, in the event that determination is made that theprocessing of all the pixels has not been completed, the processingreturns to step S3302, wherein the subsequent processing is repeatedlyperformed. That is to say, the pixels that have not become a pixel ofinterest are sequentially taken as a pixel of interest, and theprocessing in step S3302 through S3304 is repeatedly performed.

In the event that the processing of all the pixels has been completed(in step S3305, in the event that determination is made that theprocessing of all the pixels has been completed), the output pixel valuecalculation unit 3304 outputs the image in step S3306. Then, the imagegenerating processing ends.

Thus, in the above Expression (135), an integral range may be setarbitrarily, and accordingly, with the three-dimensional reintegrationmethod, it becomes possible to create a pixel having an arbitrarypowered resolution as to the original pixel (pixel of an input imagefrom the sensor 2 (FIG. 89)) without deterioration by appropriatelychanging this integral range.

That is to say, with the three-dimensional reintegration method,appropriately changing an integral range in the temporal directionenables temporal resolution to be created. Also, appropriately changingan integral range in the spatial direction enables spatial resolution tobe created. Further, appropriately changing each integral range in thetemporal direction and in the spatial direction enables both temporalresolution and spatial resolution to be created.

Specifically, with the three-dimensional reintegration method,approximation is not necessary when degenerating three dimension to twodimension or one dimension, thereby enabling high-precision processing.Also, movement in an oblique direction may be processed withoutdegenerating to two dimension. Further, no degenerating to two dimensionenables process at each dimension. For example, with the two-dimensionalreintegration method, in the event of degenerating in the spatialdirections (X direction and Y direction), process in the t directionserving as the temporal direction cannot be performed. On the otherhand, with the thee-dimensional reintegration method, any process in thetime-space directions may be performed.

Note that as described above, creation of any one of temporal resolutionand spatial resolution may be performed even with the one-dimensionalreintegration method, but creation of both temporal resolution andspatial resolution cannot be performed with the one-dimensionalreintegration method in theory, which becomes possible only byperforming two-dimensional or more reintegration. That is to say,creation of both temporal resolution and spatial resolution becomespossible only by employing the above two-dimensional reintegrationmethod and the three-dimensional reintegration method.

Also, the three-dimensional reintegration method takes notone-dimensional and two-dimensional but three-dimensional integrationeffects into consideration, and accordingly, an image more similar tothe light signal in the actual world 1 (FIG. 89) may be created.

Next, with the signal processing device 4 shown in FIG. 3, datacontinuity is detected at the data continuity detecting unit 101, andestimation of actual world 1 signal waveforms, i.e., for example, anapproximation function approximating the X-cross-section waveform F(x),is obtained at the actual world estimating unit 102, based on thecontinuity.

Thus, waveform estimation of actual world 1 signals is performed at thesignal processing device 4 based on continuity, so in the event that thecontinuity detected by the data continuity detecting unit 101 isincorrect or detection precision thereof is poor, the estimationprecision of the waveform of the actual world 1 signals is also poor.

Also, the signal processing device 4 performs signal processing based oncontinuity which actual world 1 signals, which is an image in this case,for example, have, signal processing with better precision than signalprocessing with other signal processing devices can be performed for theparts of the actual world 1 signals where continuity exists, andconsequently, an image closer to the image corresponding to the actualworld 1 signals can be output.

However, the signal processing device 4 cannot perform signal processingfor parts of the actual world 1 signals where no clear continuity existsat the same level of precision as four parts were continuity does exist,since the signal processing is being performed based on continuity, andconsequently, an image including error with regard to the imagecorresponding to the actual world 1 signals is output.

Accordingly, in order to obtain an image closer to the imagecorresponding to the actual world 1 signals with the signal processingdevice 4, processing regions where signal processing with the signalprocessing device 4 is to be performed, precision of the continuity usedwith the signal processing device 4, and so forth, are issues.

Here, FIG. 111 illustrates the configuration example of anotherembodiment of the signal processing device 4 in FIG. 1.

In FIG. 111, the signal processing device 4 comprises a processingregion setting unit 10001, a continuity setting unit 10002, an actualworld estimating unit 10003, an image generating unit 10004, an imagedisplay unit 10005, and a user I/F (Interface) 10006.

With the signal processing device 4 of which the configuration is shownin FIG. 111, image data (input image) which is an example of the data 3,is input from the sensor 2 (FIG. 1), and the input image is supplied tothe processing region setting unit 10001, the continuity setting unit10002, the actual world estimating unit 10003, the image generating unit10004, and the image display unit 10005.

The processing region setting unit 10001 sets the processing region forthe input image, and supplies processing region information identifyingthe processing region to the continuity setting unit 10002, the actualworld estimating unit 10003, and the image generating unit 10004.

The continuity setting unit 10002 recognizes the processing region inthe input image from the processing region information supplied from theprocessing regions setting unit 10001, sets the continuity of the actualworld 1 signals lost from the image data at that processing region, andsupplies continuity information representing that continuity to theactual world estimating unit 10003 and the image generating unit 10004.

The actual world estimating unit 10003 is configured of a modelgenerating unit 10011, equation generating unit 10012, and actual worldwaveform estimating unit 10013, and estimates the actual world 1 signalsfrom the image data within the processing region, according to thecontinuity of the corresponding actual world 1 signals.

That is to say, the model generating unit 10011 recognizes theprocessing region in the input image from the processing regioninformation supplied from the processing region setting unit 10001,generates a function serving as a model which models the relationbetween the pixel values of the pixels within the processing region andthe actual world 1 signals, according to the pixels making up theprocessing region and the continuity of the actual world 1 signalscorresponding to the image data in the processing region, and suppliesthis to the equation generating unit 10012.

The question generating unit 10012 recognizes the processing region inthe input image from the processing region information supplied from theprocessing regions setting unit 10001. Further, the equation generatingunit 10012 substitutes the pixel values of each of the pixels making upthe processing region into the function serving as a model that has beensupplied from the model generating unit 10011, thereby generating anequation, which is supplied to the actual world waveform estimating unit10013.

The actual world waveform estimating unit 10013 estimates the waveformof the actual world 1 signals, by computing the equation supplied fromthe equation generating unit 10012. That is to say, the actual worldwaveform estimating unit 10013 obtains an approximation functionapproximating the actual world 1 signals by solving the equationsupplied from the equation generating unit 10012, and supplies theapproximation function to the image generating unit 10004, as estimationresults of the waveform of the actual world 1 signals. Note thatapproximation function approximating the actual world 1 signals includefunctions with constant function values, regardless of argument value.

The image generating unit 10004 generates signals closer approximatingthe actual world 1 signals, based on the approximation functionrepresenting the waveform of the actual world 1 signals estimated at theactual world estimating unit 10003, and the continuity informationsupplied from the continuity setting unit 10002. That is to say, theimage generating unit 10004 recognizes the processing region in theinput image from the processing region information supplied from theprocessing region setting unit 10001, and generates image data closerapproximating the image corresponding to the actual world 1 signals withregard to the processing region, based on the approximation functionsupplied from (the actual world waveform estimating unit 10013) of theactual world estimating unit 10003, and the continuity informationsupplied from the continuity setting unit 10002.

Further, the image generating unit 10004 synthesizes the input image andimage data generated based on the approximation function (hereafterreferred to as approximation image as appropriate), generates an imagewherein the portion of the processing region of the input image has beensubstituted with the approximation image, and supplies the image as anoutput image to the image display unit 10005.

The image display unit 10005 is configured of a CRT (Cathode Ray Tube)or an LCD (Liquid Crystal Display), and displays input images, or outputimages supplied from the image generating unit 10004.

Note that the image display unit 10005 can be configured of single ormultiple CRTs or LCDs. In the event of configuring the image displayunit 10005 with a single CRT or LCD, an arrangement maybe made whereinthe screen of the single CRT or LCD is divided into multiple screens,with input images displayed on one screen and output images displayed onanother screen. Further, in the event of configuring the image displayunit 10005 of multiple CRTs or LCDs, an arrangement may be made whereininput images are displayed on one CRT or LCD, and outputs images aredisplayed on another CRT or LCD.

Also, the image display unit 10005 performs display of various types inaccordance with the output of the user I/F 10006. That is to say, theimage display unit 10005 displays a cursor, for example, and in theevent that the user operates the user I/F 10006 so as to move thecursor, the cursor is moved in accordance with the operations thereof.Also, in the event that the user operates the user I/F 10006 so as toselect a predetermined range, the image display unit 10005 displays aframe surrounding the range selected on the screen in accordance withthe operations thereof.

The user I/F 10006 is operated by the user, and in accordance with theuser operations, supplies information relating to at least one of, forexample, processing range, continuity, and real world signals, to theprocessing region setting unit 10001, continuity setting unit 10002, oractual world estimating unit 10003.

That is to say, the user views an input image or output image displayedon the image display unit 10005, and operates the user I/F 10006 so asto provide input with regard to the input image or output image. Theuser I/F 10006 is operated by the user, and in accordance with the useroperations, supplies information relating to processing range,continuity, or real world signals, to the processing region setting unit10001, continuity setting unit 10002, or actual world estimating unit10003, as assisting information for assisting the processing of theprocessing region setting unit 10001, continuity setting unit 10002, oractual world estimating unit 10003.

Upon assisting information being supplied from the user I/F 10006, theprocessing region setting unit 10001, continuity setting unit 10002, oractual world estimating unit 10003 each perform setting of processingregion, setting of continuity, or estimation of actual world 1 signals,based on the assisting information.

Note however, that the processing region setting unit 10001, continuitysetting unit 10002, or actual world estimating unit 10003, can eachperform the setting of processing region, setting of continuity, orestimation of actual world 1 signals, even without using the assistinginformation, i.e., even without the user operating the user I/F 10006.

Specifically, with the processing region setting unit 10001, processingcan be performed in the same way as with the data continuity detectingunit 101 shown in FIG. 3, as described with reference to FIG. 30 to FIG.48, wherein a continuity region is detected from the input image, and arectangular (oblong) region surrounding the continuity region is set asa processing region.

Also, with the continuity setting unit 10002, processing can beperformed in the same way as with the data continuity detecting unit 101shown in FIG. 3, as described with reference to FIG. 49 through FIG. 57,wherein data continuity is detected from the input image, and thecontinuity of the corresponding actual world 1 signals is set based onthe continuity of that data, i.e., for example, the data continuity isset as continuity of the actual world 1 signals with no change.

Further, with the actual world estimating unit 10003, processing may beperformed in the same way as with the actual world estimating unit 102shown in FIG. 3, as described with reference to FIG. 58 through FIG. 88,where actual world 1 signals are estimated from the image data of theprocessing region set by the processing region setting unit 10001,corresponding to the continuity set by the continuity setting unit10002. Note that while the data continuity has been used for estimatingactual world 1 signals that the actual world estimating unit 102 in FIG.3, an arrangement may be made wherein the continuity of correspondingactual world 1 signals is used instead of the data continuity forestimating actual world 1 signals.

Next, the processing of the signal processing device 4 shown in FIG. 111will be described with reference to the flowchart in FIG. 112.

First, in step S10001, the signal processing device 4 performspre-processing, and the flow proceeds to step S10002. That is to say,the signal processing device 4 supplies one frame or one field, forexample, of the input image, supplied from the sensor 2 (FIG. 1) as data3, to the processing region setting unit 10001, continuity setting unit10002, actual world estimating unit 10003, image generating unit 10004,and image display unit 10005. Further, the signal processing unit 4causes the image display unit 10005 to display the input image.

In step S10002, the user I/F 10006 determines whether or not there hasbeen some sort of user input, by the user operating the user I/F 10006.In step S10002, in the event that determination is made that there is nouser input, i.e., in the event that the user has made no operations,step S10003 through step S10005 are skipped, and the flow proceeds tostep S10006.

Also, in the event that determination is made in step S10002 that therehas been user input, i.e., in the event that the user has viewed theinput image displayed on the image display unit 10005 and operated theuser I/F 10006, thereby making user input indicating some sort ofinstruction or information, the flow proceeds to step S10003, where theuser I/F 10006 determines whether or not the user input is endinginstructions for instructing ending of the processing of the signalprocessing device 4.

In the event that determination is made in step S10003 that the userinput is ending instructions, the signal processing device 4 endsprocessing.

Also, in the event that determination is made in step S10003 that theuser input is not ending instructions, the flow proceeds to step S10004,where the user I/F 10006 determines whether or not the user input isassisting information. In the event that determination is made in stepS10004 that the user input is not assisting information, the flow skipsstep S10005, and proceeds to step S10006.

Also, in the event that determination is made in step S10004 that theuser input is assisting information, the flow proceeds to step S10005,where the user I/F 10006 supplies the assisting information to theprocessing region setting unit 10001, continuity setting unit 10002, oractual world estimating unit 10006, and the flow proceeds to stepS10006.

In step S10006, the processing region setting unit 10001 sets theprocessing region based on the input image, and supplies the processingregion information identifying the processing region to the continuitysetting unit 10002, actual world estimating unit 10003, and imagegenerating unit 10004, and the flow proceeds to step S10007. Now, in theevent that assisting information has been supplied from the user I/F10006 in that immediately-preceding step S10005, the processing regionsetting unit 10001 performs setting of the processing region using thatassisting information.

In step S10007, the continuity setting unit 10002 recognizes theprocessing region in the input image, from the processing regioninformation supplied from the processing region setting unit 10001.Further, the continuity setting unit 10002 sets continuity of the actualworld 1 signals that has been lost in the image data of the processingregion, and supplies continuity information indicating the continuitythereof to the actual world estimating unit 10003, and the flow proceedsto step S10008. Now, in the event that assisting information has beensupplied from the user I/F 10006 in that immediately-preceding stepS10005, the continuity setting unit 10002 performs setting of continuityusing that assisting information.

In step S10008, the actual world estimating unit 10003 estimates actualworld 1 signals regarding to the image data with in the processingregion of the input image, according to the continuity of thecorresponding actual world 1 signals.

That is to say, at the actual world estimating unit 10003, the modelgenerating unit 10011 recognizes the processing region in the inputimage, from the processing region information supplied from theprocessing region setting unit 10001, and also recognizes continuity ofthe actual world 1 signals corresponding to the image data in theprocessing region, from the continuity information supplied from thecontinuity setting unit 10002. Further the model generating unit 10011generates a function serving as a model modeling the relation betweenthe pixel values of each of the pixels within the processing region andthe actual world 1 signals, according to the pixels making up theprocessing region in the input image, and the continuity of actual world1 signals corresponding to the image data of the processing region, andsupplies this to the equation generating unit 10012.

The equation generating unit 10012 recognizes the processing region inthe input image from the processing region information supplied from theprocessing region setting unit 10001, and substitutes the pixel valuesof each of the pixels of the input image making up the processing regioninto the function serving as the model which is supplied from the modelgenerating unit 10011, thereby generating an equation for obtaining anapproximation function approximating the actual world 1 signals, whichis supplied to the actual world waveform estimating unit 10013.

The actual world waveform estimating unit 10013 estimates the waveformof the actual world 1 signals by computing the equation supplied fromthe equation generating unit 10012. That is, the actual world waveformestimating unit 10013 obtains the approximation function serving as amodel modeling the actual world 1 signals by solving the equationsupplied from the equation generating unit 10012, and supplies theapproximation function to the image generating unit 10004 as estimationresults of the waveform of the actual world 1 signals.

Note that with the actual world estimating unit 10003, in the event thatassisting information has been supplied from the user I/F 10006 in theimmediately-preceding step S10005, at the model generating unit 10011and equation generating unit 10012, processing is performed using thatassisting information.

Following processing of step S10008, the flow proceeds to step S10009,where the image generating unit 10004 generates signals closerapproximating the actual world 1 signals based on the approximationfunction approximating the waveform of the actual world 1 signalssupplied from (the actual world waveform estimating unit 10013 of) theactual world estimating unit 10003. That is to say, the image generatingunit 10004 recognizes the processing region in the input image from theprocessing region information supplied from the processing regionsetting unit 10001, and with regard to this processing region, generatesan approximation image which is image data closer approximating theimage corresponding to the actual world 1 signals, based on theapproximation function supplied from the actual world estimating unit10003. Further the image generating unit 10004 generates, as an outputimage, an image wherein the portion of the processing region of theinput image has been replaced with the approximation image, and suppliesthis to the image display 10005, and the flow proceeds from step S10009to step S10010.

In step S10010, the image display unit 10005 displays the output imagesupplied from the image generating unit 10004 instead of the input imagedisplayed in step S10001, or along with the input image, and the flowproceeds to step S10011.

In step S10011, the user I/F 10006 determines whether or not there hasbeen user input of some sort by the user operating the user I/F 10006,in the same way as with step S10002, and in the event that determinationis made that there has been no user input, i.e., in the event that theuser has made no operations, the flow returns to step S10011, and awaitsuser input.

Also, in the event that determination is made in step S10011 that therehas been user input, i.e., in the event that the user has viewed theinput image or output image displayed on the image display unit 10005and operated the user I/F 10006, thereby making some sort of user inputindicating instruction or information, the flow proceeds to step S10012,where the user I/F 10006 determines whether or not the user input isending instructions instructing ending of the processing of the signalprocessing device 4.

In the event that determination is made in step S10012 that the userinput is ending instructions, the signal processing device 4 endsprocessing.

Also, in the event that determination is made in step S10012 that theuser input is not ending instructions, the flow proceeds to step S10013,where the user I/F 10006 determines whether or not the user input isassisting information. In the event that determination is made in stepS10013 that the user input is not assisting information, the flowreturns to step S10011, and the same processing is repeated thereafter.

Also, in step S10013, in the event that determination is made that theuser input is assisting information, the flow returns to step S10005,and as described above, the user I/F 10006 supplies the assistinginformation to the processing region setting unit 10001, the continuitysetting unit 10002, or the actual world estimating unit 10006. The flowthen proceeds from step S10005 to step S10006, and hereafter the sameprocessing is repeated.

As described above, with the signal processing device 4 shown in FIG.111, assisting information for assisting the processing region settingunit 10001, the continuity setting unit 10002, or the actual worldestimating unit 10006, is supplied from the user I/F 10006 to theprocessing region setting unit 10001, the continuity setting unit 10002,or the actual world estimating unit 10003 and processing region setting,continuity setting, or actual world 1 signals estimation is performed atthe processing region setting unit 10001, the continuity setting unit10002, or the actual world estimating unit 10003 based on the assistinginformation from the user I/F 10006, thereby improving the processingprecision of the processing region setting unit 10001, the continuitysetting unit 10002, or the actual world estimating unit 10003, andenabling, for example, a high-quality output image meeting userpreferences.

Next, various types of application examples of the signal processingdevice 4 shown in FIG. 111 will be described.

FIG. 113 illustrates a configuration example of an embodiment of anapplication of the signal processing device 4 shown in FIG. 111.

An image of, for example, one frame or one field as the data 3 from thesensor 2 is input into the signal processing device 4 in FIG. 113. Here,let us say that an object having a predetermined shape is displayed inthe input image, such object moving in the horizontal direction(crosswise direction) of the input image at a constant speed of v pixelsper shutter time (exposure time). In other words, regarding the inputimage, the object is moving in the horizontal direction at the movementamount of v pixels, and therefore because of the time integration effectof the sensor 2, the light signal of the object and the light signal ofthe portion other than the object are mixed (time mixture), and thus,the image is blurred at portions such as the borders of the object. Withthe signal processign device 4 shown in FIG. 113, a high-quality outputimage is generated, wherein movement blurring due to such time mixturehas been elimited from the input image.

In FIG. 113, the processing region setting unit 15011, the continuitysetting unit 15012, the actual world estimating unit 15013, imagegenerating unit 15014, image display unit 15015, and user I/F 15016,each correspond to the processing region setting unit 10001, thecontinuity setting unit 10002, the actual world estimating unit 10003,image generating unit 10004, image display unit 10005, and user I/F10006 shown in FIG. 111, and basically perform the same processing asthe processing region setting unit 10001, the continuity setting unit10002, the actual world estimating unit 10003, image generating unit10004, image display unit 10005, and user I/F 10006. Further, in FIG.113, the actual world estimating unit 15013 comprises a model generatingunit 15021, an equation generating unit 15022, and actual world waveformestimating unit 15023. The model generating unit 15021, equationgenerating unit 15022, and actual world waveform estimating unit 15023correspond to each of the model generating unit 10011, equationgenerating unit 10012, and actual world waveform estimating unit 10013in FIG. 111, and basically perform the same functions as each of themodel generating unit 10011, equation generating unit 10012, and actualworld waveform estimating unit 10013.

Note however, that in FIG. 113, the assisting information which the userI/F 15016 outputs by the user operating the user I/F 15016, is suppliedonly to the equation generating unit 15022.

In other words, in FIG. 113, by operating the user I/F 15016, with theequation generating unit 15022, the user can set the constraintconditions which constrain the relation between each of the pixels of animage with no movement blurring, which is an image of the actual world 1corresponding to the input image, and when the user performs anoperation to set the constraint conditions, the user I/F 15016 suppliesthe constraint condition information expressing the constraintconditions set by the operation to the equation generating unit 15022.

Specifically, for example, a user who views an image displayed on theimage display unit 15015 reflecting a blurring image guesses theoriginal image, that is to say, an image without movement blurring whichis an image of the actual world 1 corresponding to the input image.Also, for example, the user guesses predetermined regions where blurringis occurring as the edge portions, and guesses the relation of the pixelvalues between each of the pixels (the constraint conditions), such asthe difference being large between the pixel values (level) between theadjacent pixels at the edge portions. Further, the user can set theconditions suitable for the guessed relation of the pixels value betweeneach of the pixels into the equation generating unit 15022, by operatingthe user I/F 15016.

Also, in FIG. 113, the processing region information, which is suppliedfrom the processing region setting unit 15011 to the actual worldestimating unit 15013, is supplied to the model generating unit 15021and the equation generating unit 15022 of the actual world estimatingunit 15013. The model generating unit 15021 and the equation generatingunit 15022 recognize the processing region in the input image from theprocessing region information which is supplied from the processingregion setting unit 15011.

The continuity setting unit 15012 sets a movement amount (movementvector) in the input image as the continuity information that an objectis moving in the horizontal direction of the image at the constant speedof v pixels per shutter time, and supplies this movement amount v as thecontinuity information to the model generating unit 15021 of the actualworld estimating unit 15013.

The model generating unit 15021 generates an equation (hereafterreferred to as model equation as appropriate) as a model (hereafterreferred to as relation model as appropriate), modeling the relationbetween the signal of the actual world 1 and the pixel values of each ofthe pixels of the movement blurring image as the input image whichconsiders the movement amount v as the continuity information suppliedfrom the continuity setting unit 15012, regarding the processing regionrecognized by the processing region information which is supplied fromthe processing region setting unit 15011, and supplies the generatedmodel equation to the equation generating unit 15022.

The equation generating unit 15022 generates a constraint conditionExpression (hereafter referred to as constraint condition expression asappropriate), based on the constraint condition information suppliedfrom the user I/F 15016, and substitutes the pixel values of each of thepixels of the input image comprising the processing region into thisconstraint condition expression and the model equation which is suppliedfrom the model generating unit 15021 to the equation generating unit15022, and thus, generates an equation to find the approximationfunction as a model (hereafter referred to as approximation model)wherein the signal of the actual world 1 is modeled, and supplies thisto the actual world waveform estimating unit 15023.

The actual world waveform estimating unit 15023 estimates the waveformof the signal of the actual world 1, by computing the equation suppliedfrom the equation generating unit 15022. In other words, the actualworld waveform estimating unit 15023 finds the approximation function asan approximation model, and supplies the approximation function to theimage generating unit 15014 as the estimation result of the signalwaveform of the actual world 1. Here, the approximation function whichapproximates the signal of the actual world 1 has contained a functionvalue that is a fixed function, regardless of the argument value.

The image generating unit 15014 recognizes the processing region of theinput image, based on the processing region information supplied fromthe processing region setting unit 15011. Also, the image generatingunit 15014 generates a signal more closely approximating the signal ofthe actual world 1, from the approximation function supplied from theactual world waveform estimating unit 15023, that is to say, generatesan image with no movement blurring, and replaces the image of theprocessing region of the input image with an image wherein the image hasno movement blurring, generates this as an output image, and suppliesthis to the image display unit 15015.

Next, the processing of the signal processing device 4 in FIG. 113 willbe described with reference to the flowchart in FIG. 114.

First, in step S15001, the signal processing device 4 performspre-processing, and the flow proceeds to step S15002. That is to say,the signal processing device 4 supplies one frame or one field, forexample, of the input image, supplied from the sensor 2 (FIG. 1) as data3, to the processing region setting unit 15011, continuity setting unit15012, actual world estimating unit 15013, image generating unit 15014,and image display unit 15015. Further, the signal processing unit 4causes the image display unit 15015 to display the input image.

In step S15002, the user I/F 15016 determines whether or not there hasbeen some sort of user input, by the user operating the user I/F 15016.In step S15002, in the event that determination is made that there is nouser input, i.e., in the event that the user has made no operations,step S15003 through step S15005 are skipped, and the flow proceeds tostep S15006.

On the other hand, in step S15002, in the event that determination ismade that there has been user input, i.e., in the event that the userhas viewed the input image displayed on the image display unit 15015 andoperated the user I/F 15016, thereby making user input indicating somesort of instruction or information, the flow proceeds to step S15003,where the user I/F 15016 determines whether or not the user input isuser instructions instructing ending of the processing of the signalprocessing device 4.

In the event that determination is made in step S15003 that the userinput is ending instructions, the signal processing device 4 ends theprocessing.

Also, in the event that determination is made in step S15003 that theuser input is not ending instructions, the flow proceeds to step S15004,where the user I/F 15016 determines whether or not the user input isconstraint condition information. In the event that determination ismade in step S15004 that the user input is not constraint conditioninformation, step S15005 is skipped, and the flow proceeds to stepS15006.

Also, in the event that determination is made in step S15004 that theuser input is constraint condition information, the flow proceeds tostep S15005, where the user I/F 15016 supplies the constraint conditioninformation to the equation generating unit 15022, and the flow proceedsto step S15006.

In step S15006, the processing region setting unit 15011 sets theprocessing region based on the input image, and supplies the processingregion information identifying the processing region to the continuitysetting unit 15012, the model generating unit 15021 and the equationgenerating unit 15022 of the actual world estimating unit 15013, andimage generating unit 15014, and the flow proceeds to step S15007. Now,the setting of the processing region in step S15006 can be arranged bythe user inputting the processing region instruction information byoperating the user I/F 15016, and can be performed based on thisprocessing region instruction information, or can be performed with noprocessing region instruction information from the user.

In step S15007, the continuity setting unit 15012 recognizes theprocessing region in the input image, from the processing regioninformation supplied from the processing region setting unit 15011.Further, the continuity setting unit 15012 sets the continuity of thesignal of the actual world 1, of which one portion of continuity hasbeen lost of the image data of the processing region, and the continuityinformation showing this continuity is supplied to the model generatingunit 15021 of the actual world estimating unit 15013, and the flowproceeds to step S15008. Here the continuity setting unit 15012 sets themovement amount v, expressing that the movement is in the horizontaldirection at a constant speed of v pixels per shutter time (exposuretime), as the continuity information, and supplies this to the modelgenerating unit 15021 of the actual world estimating unit 15013. Here,the continuity setting unit 15012 sets the movement amount showing onlythe movement size as continuity information, under the premise that theobject is moving in a horizontal direction, but a movement vectorshowing the size and direction of the movement of the object can be setas continuity information as well. The setting of continuity in stepS15007 can be arranged by the user inputting the continuity instructioninformation by operating the user I/F 15016, and can be performed basedon this continuity instruction information, or can be performed with nocontinuity instruction information from the user.

The actual world estimating unit 15013 performs actual world estimatingprocessing in step S15008. In other words, with the actual worldestimating unit 15013, the model generating unit 15021 generates anequation (model equation) as the model (relation model) wherein therelation of the pixel values of each of the pixels of the input imagewhich has movement blurring and the signal of the actual world 1 ismodeled, based on the movement amount v supplied from the continuitysetting unit 15012 in step S15007 and the processing region informationsupplied from the processing region setting unit 15011 in step S15006,and the generated model is supplied to the equation generating unit15022.

The equation generating unit 15022 generates a constraint conditionexpression based on the constraint condition information supplied fromthe user I/F 15016 in step S15005, and further, generates an equationfor finding the approximation function of the model (approximationmodel) wherein the signal of the actual world 1 is modeled, from theconstraint condition expression and the model equation supplied from themodel generating unit 15021 to the equation generating unit 15022, andsubstitutes the pixel values of each of the pixels of the input imageinto this equation, and supplies this to the actual world waveformestimating unit 15023.

The actual world waveform estimating unit 15023 estimates the waveformof the signal of the actual world 1, that is to say, finds theapproximation function as a model which models the signal of the actualworld 1, by calculating the equation supplied from the equationgenerating unit 15022, and supplies this approximation function to theimage generating unit 15014 as the estimation result of the signalwaveform of the actual world 1.

The actual world estimating processing of step S15008 will be describedin detail later with reference to FIG. 130.

After the processing in step S15008, the flow proceeds to step S15009,and the image generating unit 15014 generates a signal more closelyapproximating the signal from the actual world 1, from the approximationfunction supplied from the actual world waveform estimating unit 15023,that is to say, generates an image with no movement blurring, and basedon the processing region information supplied from the processing regionsetting unit 15011, replaces the portion of the processing region of theinput image with an image which has no movement blurring generated,generates this as an output image, and supplies this to the imagedisplay unit 15015, and the flow proceeds to step S15010.

In step S15010, the image display unit 15015 displays the output imagesupplied from the image generating unit 15014 instead of the input imagedisplayed in step S15001, or along with the input image, and the flowproceeds to step S15011.

In step S15011, as with the case of step S15002, the user I/F 12006determines whether or not there has been some sort of user input by theuser operating the user I/F 15016, and in the event that determinationis made that there has been no user input, i.e., in the event that theuser has made no operations, the flow returns to step S15011, and awaitssome sort of user input.

Also, in the event that determination is made in step S15011 that therehas been user input, i.e., in the event that the user has viewed theinput image or output image displayed on the image display unit 15015and operated the user I/F 15016, thereby making user input representingsome sort of instructions or information, the flow proceeds to stepS15012, where the user I/F 15016 determines whether or not the userinput is ending instructions instructing ending of the processing of thesignal processing device 4.

In the event that determination is made in step S15012 that the userinput is ending instructions, the signal processing device 4 endsprocessing.

On the other hand, in the event that determination is made in stepS15012 that the user input is not ending instructions, the flow proceedsto step S15013, where the user I/F 15016 determines whether or not theuser input is constraint condition information. In the event thatdetermination is made in step S15013 that the user input is notconstraint condition information, the flow returns to step S15011, andthe same processing is repeated thereafter.

Also, in step S15013, in the event that determination is made that theuser input is constraint condition information, the flow returns to stepS15005, and as described above, the user I/F 15016 supplies theconstraint condition information to the equation generating unit 15022.The flow then proceeds from step S15005 to step S15006, and hereafterthe same processing is repeated.

Now, in the event that the signal processing device 4 repeats theprocessing of step S15005 through step S15013, the processing of theabove-described step S15006 and S15007 are the same as that set by thefirst processing S15006 and S15007, and the same process as the firstprocess may be repeated, or may be skipped.

Thus, with the signal processing device 4 in FIG. 113, by finding anapproximation function as a model wherein the signal of the actual world1 is modeled, an output image of high image quality with no movementblurring can be generated.

Also, by operating the user I/F 15016, the user can set constraintconditions which constrain the relation between each of the pixels inthe original image having no movement blurring, and in the event that anew constraint condition is input, an image without movement blurringcan be requested again, and therefore an output image of high imagequality according to the preference of the user can be easily acquired.

With the signal processing device 4 in FIG. 113, an output image withoutmovement blurring is generated by finding an approximation functionwhich approximates the light signal of the actual world 1, but thisoutput image can be viewed as having the movement blurring removed fromthe input image. Accordingly, it can be said that processing isperformed for movement blurring removal with the signal processingdevice 4 in FIG. 113.

The processing for movement blurring removal with the signal processingdevice 4 in FIG. 113 will be described in detail below. First, bymodeling (formulating) the mechanism for movement blurring generation, arelation model is generated which models the relation between the pixelvalues of each of the pixels of the input image with movement blurringand the signal of the actual world 1. The light signal of the actualworld 1 has no movement blurring, and therefore the relation model is amodeling of the relation between the pixel values of each of the pixelsof the movement blurring image and the pixel values of each of thepixels of the image with no movement blurring.

FIG. 115 is a diagram describing the light signal of the actual world 1.

The light signal of the actual world 1 is to be regarded as a waveformF(x,y) on an X-Y plane (a plane formed of the X direction which is onedirection of a spatial direction, and a Y direction which isperpendicular to the X direction) as illustrated in FIG. 115. The signalinput into the signal processing device 4 of FIG. 113 is moving in thehorizontal direction at the movement amount of v pixels as describedabove, and therefore is the same value as the waveform F(x,y) which ismoving at the movement amount of v pixels per shutter time in the samedirection as the spatial direction X. Hereafter, the waveform F(x,y)will be called the light signal function F(x,y).

The actual world estimating unit 15013 of the signal processing device 4in FIG. 113 estimates a light signal function F(x,y) of the actual world1 by finding an approximation function f(x,y) which approximates thelight signal function F(x,y) of the actual world 1, based on the inputimage from the sensor 2.

The input image which is the output of the sensor 2 is the image dataconfigured with multiple pixels wherein the above-described light signalof the actual world 1 (the light signal function F(x,y)) is projected aspixel values from the detecting elements (pixels) of the sensor whichhas an integration effect.

The integration effect in the case that the sensor 2 is a CCD will bedescribed with reference to FIG. 116.

As shown in FIG. 116, multiple detecting elements (pixels) 15031 aredisposed on the plane of the sensor 2.

In the example shown in FIG. 116, the direction parallel to 1predetermined side of the detecting element 15031 is the X directionwhich is one direction in the spatial direction, and the directionperpendicular to the X direction is the Y direction, which is the otherdirection in the spatial directions. The direction perpendicular to theX-Y plane is the t direction, which is the time direction.

Also, with the example in FIG. 116, the spatial shape of the respectivedetecting elements 15031 (corresponding to the pixels of the image dataoutput by the sensor 2) of the sensor 2 is taken as a square of whichone side is 1 in length. The shutter time (exposure time) of the sensor2 is taken as 1.

Further, with the example in FIG. 116, the center of one detectingelement 15031 of the sensor 2 is taken as the origin (position x=0 inthe X direction, and position y=0 in the Y direction) in the spatialdirection (X direction and Y direction), and the intermediatepoint-in-time of the exposure time is taken as the origin (position t=0in the t direction) in the temporal direction (t direction).

In this case, the detecting element 15031 of which the center is in theorigin (x=0, y=0) in the spatial direction subjects the light signalfunction F(x, y, t) to integration with a range between −0.5 and 0.5 inthe X direction, range between −0.5 and 0.5 in the Y direction, andrange between −0.5 and 0.5 in the t direction, and outputs the integralvalue thereof as a pixel value P.

That is to say, the pixel value P output from the detecting element15031 of which the center is in the origin in the spatial direction isrepresented with the following Expression (137).

$\begin{matrix}{P = {\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{\int_{- 0.5}^{+ 0.5}{{F\left( {x,y,t} \right)}{\mathbb{d}x}{\mathbb{d}y}{\mathbb{d}t}}}}}} & (137)\end{matrix}$

The other detecting elements 15031 also output the pixel value P shownin Expression (137) by taking the center of the subject detectingelement 15031 as the origin in the spatial direction in the same way.

Thus, the input image from the sensor 2 is a collection of data whereineach of the detecting elements (pixels) 15031 has a constant pixel valuedue to the integration effect of the sensor 2. Also, the integrationeffect of the sensor 2 comprises the integration of the spatialdirection (X-Y direction) of the light signal of the actual world 1(special mixture) and the integration in the time direction (tdirection) (time mixture).

Accordingly, even in the cases wherein movement blurring of an object isoccurring in an input image, strictly speaking the pixel value of eachof the pixels of the input image is the value generated by both thespatial integration effect and the time integration effect (spatialmixture and time mixture have occurred), but with the movement blurringdue to the movement of an object in the horizontal direction, the effectof the time integration effect is great, and therefore with thisembodiment, in order to model the generation of movement blurring moreeasily, let us say that the approximation function f(x,y) is modeled ashaving generated only the time integration effect by the object moving vpixels per shutter time.

In other words, the approximation function f(x,y) approximating theactual world 1 has no spatial integration effect, as illustrated in FIG.117, that is to say, is a function having a constant value within thevarious pixels of the sensor, and let us say that pixel values of thevarious pixels of the input image with movement blurring occurring aretime-integrated values by the approximation function f(x,y) in FIG. 117moving in the horizontal direction at a constant speed at the movementamount v.

FIG. 118 is a diagram illustrating the relation of the input image whichis input in the signal processing device 4 and the processing regionwhich is set in the processing region setting unit 15011.

The image 15041 (input image) which is input from the sensor 2 (FIG. 1)into the signal processing device 4 is an image having a predeterminednumber of pixels in the spatial direction of the X direction and thespatial direction of the Y direction.

A region 15042 which is a portion or all of the image 15041 is set asthe processing region by the process region setting unit 15011. Theinformation illustrating the region 15042 within the image 15041 issupplied as processing region information from the processing regionsetting unit 15011 to the continuity setting unit 15012, the actualworld estimating unit 15013, and the image generating unit 15014.Hereafter the region 15042 will be called the processing region 15042.

FIG. 119 is a diagram illustrating the processing region 15042.

The processing region 15042 is a region having a processing region widthof l=10 pixels and line number of k=4, for example, as illustrated inFIG. 119. In other words, in FIG. 119, the processing region 15042 iscomprised of 10×4=40 pixels.

FIG. 120 is a diagram illustrating the approximation function f(x,y) inFIG. 117 at the timing of a predetermined instant regarding one linewherein the value of y is the predetermined value y_(c) (y=y_(c)),within the processing region 15042. In other words, FIG. 120 is adiagram illustrating the approximation function f(x, y_(c)) at thetiming of a predetermined instant. This approximation function f(x,y_(c)) is an approximation function f(x, y) illustrating the relationpixel level and the position of the special direction X when the timedirection t and the spatial direction Y is at a fixed value, and thiswill simply be called an approximation function f(x).

The approximation function in FIG. 120 can be expressed as the followingExpression (138).

$\begin{matrix}{{f(x)} = \left\{ \begin{matrix}Q_{0} & {{- 0.5} \leqq x < 0.5} \\Q_{1} & {0.5 \leqq x < 1.5} \\Q_{2} & {1.5 \leqq x < 2.5} \\Q_{3} & {2.5 \leqq x < 3.5} \\Q_{4} & {3.5 \leqq x < 4.5} \\Q_{5} & {4.5 \leqq x < 5.5} \\Q_{6} & {5.5 \leqq x < 6.5} \\Q_{7} & {6.5 \leqq x < 7.5} \\Q_{8} & {7.5 \leqq x < 8.5} \\Q_{9} & {8.5 \leqq x < 9.5}\end{matrix} \right.} & (138)\end{matrix}$

Expression (138) is an approximation function f(x) at the timing of aninstant, and actually, the approximation function f(x) has a movementamount v, and therefore after a 1/v time in FIG. 120, becomes such asthat illustrated in FIG. 121. The approximation function in FIG. 120 canbe expressed as the following Expression (139).

$\begin{matrix}{{f(x)} = \left\{ \begin{matrix}0 & {{- 0.5} \leqq x < 0.5} \\Q_{0} & {0.5 \leqq x < 1.5} \\Q_{1} & {1.5 \leqq x < 2.5} \\Q_{2} & {2.5 \leqq x < 3.5} \\Q_{3} & {3.5 \leqq x < 4.5} \\Q_{4} & {4.5 \leqq x < 5.5} \\Q_{5} & {5.5 \leqq x < 6.5} \\Q_{6} & {6.5 \leqq x < 7.5} \\Q_{7} & {7.5 \leqq x < 8.5} \\Q_{8} & {8.5 \leqq x < 9.5}\end{matrix} \right.} & (139)\end{matrix}$

Further, after a 1/v time in FIG. 121, the function becomes such as thatillustrated in FIG. 122, and the approximation function f(x) at thistime is represented with the following Expression (140).

$\begin{matrix}{{f(x)} = \left\{ \begin{matrix}0 & {{- 0.5} \leqq x < 0.5} \\0 & {0.5 \leqq x < 1.5} \\Q_{0} & {1.5 \leqq x < 2.5} \\Q_{1} & {2.5 \leqq x < 3.5} \\Q_{2} & {3.5 \leqq x < 4.5} \\Q_{3} & {4.5 \leqq x < 5.5} \\Q_{4} & {5.5 \leqq x < 6.5} \\Q_{5} & {6.5 \leqq x < 7.5} \\Q_{6} & {7.5 \leqq x < 8.5} \\Q_{7} & {8.5 \leqq x < 9.5}\end{matrix} \right.} & (140)\end{matrix}$

Here the movement amount v of the object can be considered as asimplified form as illustrated in FIG. 123, by the movement of theapproximation function f(x) illustrated in FIG. 120 through FIG. 122. Inother words, the object moves instantly to the center of the pixel nextover in the x direction every 1/v time, and stays there for the period1/v time. Accordingly, during the 1/v time, the pixel value of the samevalue as when the object is not moving as accumulated in the pixels ofthe sensor. Also, during the next 1/v time, the pixel values that arethe same as when the object is not moving, is accumulated as to thepixel that was advanced only one pixel worth in the x direction.

In other words, in the event that the object (the light signal of theactual world 1) is moving at the movement amount v, each pixel can beconsidered to have 1/v times the pixel value as the same pixel valuewhen the object is not moving at the movement amount v, accumulated inorder, for every 1/v time.

As described above, even if we think of the object as moving only onepixel worth in the x direction instantly every 1/v time, the object canbe seen as moving at the same speed in the x direction with the movementamount of v pixels for each unit of time, when viewed at a macro level,as illustrated by a straight line in FIG. 123.

The accumulation by the detecting elements of the sensor (timeintegration) of the object having the movement amount v will be furtherdescribed with reference to FIG. 124 through FIG. 126.

FIG. 124 is a diagram showing the approximation function f(x) of y=y_(c)at a predetermined precise timing, illustrated in FIG. 120, with a X-tplane.

In FIG. 124, the width of the X direction of each pixel 15031 is 1, asdescribed in FIG. 116, and the height in the t-direction is the shuttertime (=1). Also, the area (the width in the X direction x the height inthe t-direction) of each pixel 15031 are each the pixel values Q₀through Q₉ accumulated (expressed by the total amount of electriccharge) in each pixel 15031. These pixel values Q₀ through Q₉ are thepixel values of the image with no movement blurring.

On the other hand, FIG. 125 is a diagram illustrating the pixel values,which are time-integrated and accumulated, when the object is moving atthe movement amount v so as to correspond to the pixel values Q₀ throughQ₉ in FIG. 124, as P₀ through P₉. In other words this is a diagram of anX-t plane of the pixel values P₀ through P₉ of the input image,regarding one line of the y=y_(c) within the processing region 15042.

For example, based on the movement of the object described in theabove-described FIG. 120 through FIG. 123 with v=4 (pixels), when thepixel values P₀ through P₉ of FIG. 125 are expressed using the pixelvalues Q₀ through Q₉ in FIG. 124, the result is such as that in FIG.126.

In other words, for the first 1/v time of the shutter time, the pixelvalues (the electric charge corresponding thereto) equivalent to 1/v ofthe pixel value Q_(h) (h=0 through ₉) with no movement blurring areaccumulated into each of the pixels where x=h. Then, for the next 1/vtime, the pixels move over one pixel worth of the movement directionside in the X direction (the pixel on the right side in the diagram),and the pixel values equivalent to 1/v of the same pixel value Q_(h)(h=0 through 8) are accumulated into each of the pixels where x=h.Hereafter, similarly, for each 1/v time, as the pixels move over onepixel worth of the movement direction side in the X direction, the pixelvalues equivalent to 1/v of the pixel value Q_(h) (h=0 through 8) areaccumulated. Hereafter, as long as the variable h which is a referencenumeral has no special restriction, the variable h will express aninteger where h=0 through 9.

In other words, with the pixel wherein X=0 is the center thereof(hereafter, this will be called a pixel with X=0), 1/v of the pixelvalue of the pixel value Q₀ (Q₀/v) is accumulated. With the pixelwherein X=1 is the center to the immediate right of the pixel whereinX=0 (hereafter this will be called the X=1 pixel), the pixel value(Q₁/v) which is 1/v of the pixel value Q₁ and the pixel value (Q₁/v)which is 1/v of the pixel value Q₀ are accumulated. With the pixelwherein X=2 is the center to the immediate right of the pixel whereinX=1 (hereafter this will be called the X=2 pixel), the pixel value(Q₀/v) which is 1/v of the pixel value Q₀, the pixel value (Q₁/v) whichis 1/v of the pixel value Q₁, and the pixel value (Q₂/v) which is 1/v ofthe pixel value Q₂ are accumulated. Thereafter, the pixel values whichare 1/v of the pixel values Q_(h) are similarly moved over each 1/v timefor each pixel, and accumulated.

The following Expression (141) is acquired when expressing the relationillustrated in FIGS. 125 and 126 with an expression.

$\begin{matrix}{{\left( {Q_{0} + Q_{1} + Q_{2} + Q_{3}} \right)/v} = {{{P_{3}\left( {Q_{1} + Q_{2} + Q_{3} + Q_{4}} \right)}/v} = {{{P_{4}\left( {Q_{2} + Q_{3} + Q_{4} + Q_{5}} \right)}/v} = {{{P_{5}\left( {Q_{3} + Q_{4} + Q_{5} + Q_{6}} \right)}/v} = {{{P_{6}\left( {Q_{4} + Q_{5} + Q_{6} + Q_{7}} \right)}/v} = {{{P_{7}\left( {Q_{5} + Q_{6} + Q_{7} + Q_{8}} \right)}/v} = {{{P_{8}\left( {Q_{6} + Q_{7} + Q_{8} + Q_{9}} \right)}/v} = P_{9}}}}}}}} & (141)\end{matrix}$

Here, with the pixels wherein x=0, 1, or 2, there is a blank (unknown)region of the accumulated pixel values. Therefore, each of the P₀, P₁,and P₂ has not acquired an accumulated pixel value within the entireshutter time, and so cannot build an expression.

Expression (141) is a simultaneous equation wherein the mechanism ofmovement blurring generation is modeled, with the pixel values of eachof the pixels of the movement blurring image being considered to be theaccumulated (integrated) value wherein the pixel values of each of thepixels of an image with no movement blurring are accumulated(integrated), while moving at the movement amount v. In Expression(141), the pixel value P_(h) when the object is moving at the movementamount v is the pixel value already known of the input image to be inputinto the signal processing device 4, and the pixel values Q_(h) when theobject has no movement expresses the pixel value to be found with noblurring.

Thus, the pixel values P_(h) when the object is moving at the movementamount v can be said to be the pixel value wherein the pixel valuesQ_(h) when the object has no movement are mixed for 1/v time each (timemixture). Hereafter, the pixel value P_(h) when the object is moving atthe movement amount v is called the mixed pixel value P_(h), and thepixel values Q_(h) when the object has no movement is simply called thepixel values Q_(h). The approximation function f(x) as an approximationmodel wherein the signal of the actual world 1 is modeled is configuredwith a value of Q_(h), as expressed in Expression (139) with theabove-described example, and so Expression (141) is a model equation asa relation model wherein relation of the pixel values of each of thepixels of the movement blurring image as the input image and the signalof the actual world 1 is modeled. Also, finding the pixel value Q_(h)with no movement blurring is the same as finding the approximationfunction f(x) wherein the actual world estimating unit 15013 has modeledthe signal of the actual world 1.

The above-described simultaneous equation (141) can be built by settingthe processing region width l of the processing region 15042 and themovement amount v. However, the simultaneous equation (141) only has 7expressions, compared to the 10 variables of the pixel values Q_(h).Accordingly, it is difficult to find the pixel values Q_(h) with onlyExpression (141).

Therefore, two methods will be described for having at least the samenumber of expressions and number of variables.

The first method is a method of hypothesizing that the edge portions ofthe processing region is “flat”. That is to say, as shown in FIG. 127,this is a method of hypothesizing that the accumulated pixel value as tothe blank (unknown) region 15051 of the pixel value is the same as thepixel value (Q₀/v) of 1/v of the pixel value Q₀, which is the edgeportion of the first 1/v time.

In this case, the following Expression (142) can be newly acquired.

$\begin{matrix}{{\left( {Q_{0} + Q_{0} + Q_{0} + Q_{0}} \right)/v} = {{{P_{0}\left( {Q_{0} + Q_{0} + Q_{0} + Q_{1}} \right)}/v} = {{{P_{1}\left( {Q_{0} + Q_{0} + Q_{1} + Q_{2}} \right)}/v} = P_{2}}}} & (142)\end{matrix}$

From Expression (141) and Expression (142), the number of expressions is10 and the number of variables is 10, and so the pixel value Q_(h) canbe found.

With the processing which is described with reference to the flowchartin FIG. 114, in the event that there is no specification (setting) forconstraint conditions by the user, that is to say, in the event thatconstraint condition information of the equation generating unit 15022is not supplied form the user I/F 15016, for example the first methodmay be used. However, with the first method, the edge portions of theprocessing region are hypothesized as being “flat”, and so the imageconfigured from the pixel value Q_(h) to be found with the first methodcan have noise generated in some cases.

Thus, in the case that constraint condition information is supplied fromthe user I/F 15016 to the equation generating unit 15022, a secondmethod can be applied which is different from the first method. Thesecond method is a method for acquiring a number of expressions greaterthan the number of variables by combining Expression (141) with anExpression (hereafter referred to as a constraint condition expression),which is built that hypothesizes (constrains) the relation of the pixelvalues of the adjacent various pixels in the image with no blurring.

In general, an image often has a spatial correlation, and thereforeadjacent pixels often do not differ greatly. Thus, for example, acondition (hereafter referred to as a constraint condition) can behypothesized to say that the difference is 0 (hereafter this will becalled “adjacent pixel difference=0”) for the pixel value Q_(h) betweeneach of the pixels not generating movement blurring. This constraintcondition can be expressed with the following Expression (143).

$\begin{matrix}{{{Q_{h - 1} - Q_{h}} = {{0\mspace{31mu} h} = 1}},\ldots\mspace{14mu},\left( {l - 1} \right)} & (143)\end{matrix}$

Also, for example, a condition stating that “the pixel values arechanging smoothly” can be hypothesized. This constraint condition can beexpressed with the following Expression (144).

$\begin{matrix}{{{{- Q_{h - 1}} + {2Q_{h}} - Q_{h + 1}} = {{0\mspace{25mu} h} = 1}},\ldots\mspace{14mu},\left( {l - 2} \right)} & (144)\end{matrix}$

The user viewing an image displayed on the image display unit 15015reflecting a blurred object guesses the original image, that is to say,an image without movement blurring which is an image of the actual world1 corresponding to the input image, and can set the relation equation(constraint conditions) between the pixels which expresses the guessedstate, into the equation generating unit 15022, by operating the userI/F 15016.

With the signal processing device 4 in FIG. 113, the constraintcondition of Expression (143) is said to be set by the user (theconstraint condition information is supplied from the user I/F 15016which instructs Expression (143)). In this event, regarding the lineillustrated in FIG. 124, a constraint condition expression can beacquired with is expressed with the following Expression (145).

$\begin{matrix}{{{Q_{0} - Q_{1}} = 0}{{Q_{1} - Q_{2}} = 0}{{Q_{2} - Q_{3}} = 0}{{Q_{3} - Q_{4}} = 0}{{Q_{4} - Q_{5}} = 0}{{Q_{5} - Q_{6}} = 0}{{Q_{6} - Q_{7}} = 0}{{Q_{7} - Q_{8}} = 0}{{Q_{8} - Q_{9}} = 0}} & (145)\end{matrix}$

Expression (145) has 9 expressions, and when added to Expression (141),the number of expressions is 16, and the number of variables Q_(h) is10, and so for example, by using the least-square method, a variableQ_(h) wherein the sum of squares of error generated in the variousexpressions is the smallest, can be calculated. In other words, thepixel value Q_(h) with the movement blurring removed (with no movementblurring) may be estimated.

Specifically, if the error generated in each of the expressions inExpression (141) is e_(mi) (i=3 through 9), and the error generated ineach of the expressions in Expression (145) is e_(bj) (b=0 through 8),Expression (141) and Expression (145) can be expressed by Expression(146) and Expression (147), respectively.

$\begin{matrix}{{\left( {Q_{0} + Q_{1} + Q_{2} + Q_{3}} \right)/v} = {{P_{3} + {{e_{m\; 3}\left( {Q_{1} + Q_{2} + Q_{3} + Q_{4}} \right)}/v}} = {{P_{4} + {{e_{m\; 4}\left( {Q_{2} + Q_{3} + Q_{4} + Q_{5}} \right)}/v}} = {{P_{5} + {{e_{m\; 5}\left( {Q_{3} + Q_{4} + Q_{5} + Q_{6}} \right)}/v}} = {{P_{6} + {{e_{m\; 6}\left( {Q_{4} + Q_{5} + Q_{6} + Q_{7}} \right)}/v}} = {{P_{7} + {{e_{m\; 7}\left( {Q_{5} + Q_{6} + Q_{7} + Q_{8}} \right)}/v}} = {{P_{8} + {{e_{m\; 8}\left( {Q_{6} + Q_{7} + Q_{8} + Q_{9}} \right)}/v}} = {P_{9} + e_{m\; 9}}}}}}}}} & (146) \\{{{Q_{0} - Q_{1}} = {0 + e_{b\; 0}}}{{Q_{1} - Q_{2}} = {0 + e_{b\; 1}}}{{Q_{2} - Q_{3}} = {0 + e_{b\; 2}}}{{Q_{3} - Q_{4}} = {0 + e_{b\; 3}}}{{Q_{4} - Q_{5}} = {0 + e_{b\; 4}}}{{Q_{5} - Q_{6}} = {0 + e_{b\; 5}}}{{Q_{6} - Q_{7}} = {0 + e_{b\; 6}}}{{Q_{7} - Q_{8}} = {0 + e_{b\; 7}}}{{Q_{8} - Q_{9}} = {0 + e_{b\; 8}}}} & (147)\end{matrix}$

From Expression (146) and Expression (147), the following Expression(148) holds.

$\begin{matrix}{{\begin{bmatrix}{1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} \\1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1}\end{bmatrix}\begin{bmatrix}Q_{0} \\Q_{1} \\Q_{2} \\Q_{3} \\Q_{4} \\Q_{5} \\Q_{6} \\Q_{7} \\Q_{8} \\Q_{9}\end{bmatrix}} = {\begin{bmatrix}P_{3} \\P_{4} \\P_{5} \\P_{6} \\P_{7} \\P_{8} \\P_{9} \\0 \\0 \\0 \\0 \\0 \\0 \\0 \\0 \\0\end{bmatrix} + \begin{bmatrix}e_{m\; 3} \\e_{m\; 4} \\e_{m\; 5} \\e_{m\; 6} \\e_{m\; 7} \\e_{m\; 8} \\e_{m\; 9} \\e_{b\; 0} \\e_{b\; 1} \\e_{b\; 2} \\e_{b\; 3} \\e_{b\; 4} \\e_{b\; 5} \\e_{b\; 6} \\e_{b\; 7} \\e_{b\; 8}\end{bmatrix}}} & (148)\end{matrix}$

Hereafter, as long as the variables i and j have no specialrestrictions, the variable i will express an integer where i=3 through 9and j will express an integer where j=0 through 8.

Expression (148) is replaced with Expression (149), using a matrix A,and column vectors x, y, and e.

$\begin{matrix}{{Ax} = {y + e}} & (149)\end{matrix}$

In this case, the total sum E of the squared errors of the error e_(mi)of Expression (146) and the error e_(bj) of Expression (147) can beexpressed with the following expression.

$\begin{matrix}\begin{matrix}{E = {e}^{2}} \\{= {{\sum e_{mi}^{2}} + {\sum e_{bj}^{2}}}} \\{= {\left( {{A\; x} - y} \right)^{T}\left( {{A\; x} - y} \right)}} \\{= {{x^{T}A^{T}A\; x} - {2{x\;}^{T}A^{T}y} + {y^{T}y}}}\end{matrix} & (150)\end{matrix}$

In order to find the Q_(h) for which the total sum E of the squarederror is the smallest, the fact that the total sum E of the squarederror of the equation (150) can have 0 as the value partiallydifferentiated with the column vector x with Q_(h) as a component leadsinto the following expression.

$\begin{matrix}{\frac{\partial E}{\partial x} = {{2\left( {{A^{T}A\; x} - {A^{T}y}} \right)} = 0}} & (151)\end{matrix}$

From Expression (151), the column vector x (a column vector x with thepixel value of Q_(h) as a component) to be found can be expressed withExpression (152).

$\begin{matrix}{x = {\left( {A^{T}A} \right)^{- 1}A^{T}y}} & (152)\end{matrix}$

Now, with Expression (152), the superscript T means transposing, and thesuperscript −1 means an inverse matrix.

By substituting the column vector y, of which the mixed image valueP_(i) is a component, into Expression (152) and computing this, thepixel value Q_(h) with the movement blurring removed (with no movementblurring) can be estimated.

FIG. 128 is a diagram illustrating again the processing region 15042.

A column vector x (=pixel value Q_(h)) which can be found fromExpression (152) is a column vector x (=pixel value Q_(h)) of one lineof the y=y_(c) of the processing region 15042 illustrated in FIG. 128,and therefore in order to estimate a pixel value with no movementblurring for all pixels in the processing region 15042, the computationof Expression (152) must be repeated the same number of times k (=4) asthe line number.

In other words, as with the y=y_(c) line in the above description, ifthe one line which performs processing to find the pixel value Q_(h)with no movement blurring is called the line of interest, the actualworld estimating unit 15013 sets each of all of the lines, in order, asthe line of interest, and by performing processing to estimate the pixelvalues Q₀ through Q₉, which is the approximation function f(x) of theline of interest, an image with no movement blurring can be estimatedregarding the processing region 15042 of the input image.

FIG. 129 is a configuration example of the inside of the actual worldestimating unit 15013 in FIG. 113.

The actual world estimating unit 15013 comprises a model generating unit15021, an equation generating unit 15022, and an actual world waveformestimating unit 15023. Further, the equation generating unit 15022comprises a constraint condition setting unit 15061 and a normalequation generating unit 15062.

The movement amount v as the continuity information is supplied to themodel generating unit 15021 from the continuity setting unit 15012 ascontinuity information. The movement amount v in this example expressesthat the object is moving in the horizontal direction only v pixels pershutter time, within the processing region of the input image which isset by the processing region setting unit 15011, and for example, v=4,as with the above-described example. Also, the movement amount v may bea value which differs for each line, and in the event that the movementamount v differs for each line, k number, equal to the number of lines,of the movement amount v is supplied as continuity information from thecontinuity setting unit 15012 to the model generating unit 15021.

Also, processing region information which specifies the processingregion within the input image is also supplied to the model generatingunit 15021 from the processing region setting unit 15011. In the case ofusing a processing region with a rectangular-shaped region as theprocessing region, the processing region information can be expressed,for example, by coordinate values at opposite angles of arectangular-shaped processing region as to an absolute coordinatessystem wherein a predetermined position (pixel) of the input image isthe origin, or the center coordinate of the processing region and thewidth (horizontal direction) and the height (vertical direction) of theprocessing region, or the like.

The model generating unit 15021 acquires, for example, a processingregion width of l, and k line numbers, as illustrated in FIG. 119, fromthe processing region information supplied from the processing regionsetting unit 15011. In this example, let us say that the processingregion 15042 is rectangular-shaped, and for example, has l=10 and k=4.In this case, the processing region width of each line (the number ofpixels) are the same, but with the processing region 15042, each linecan have a different processing region width. In the case that theprocessing region width l differs for each line, the model generatingunit 15021 acquires a processing region width l of the same number (k)as the number of lines.

Also, the model generating unit 15021 generates an equation (modelequation) of a mixed pixel value P_(i) which is shown in theabove-described Expression (141), based on the movement amount vsupplied from the continuity setting unit 15012 and the processingregion width l and k number of lines acquired from the processing regioninformation, and supplies this to the normal equation generating unit15062.

The constraint condition information specified by the user operating theuser I/F 15016 is supplied to the constraint condition setting unit15061 from the user I/F 15016. For example, the user performs anoperation with the user I/F 15016 to select (specify) the desiredconstraint condition from the constraint conditions which constrain therelation of the pixel values Q_(h) of each of the pixels of an imagewith no movement blurring, such as the above-described Expression (143)or Expression (144), which have been set in advance inside theconstraint condition setting unit 15061, and the user I/F 15016 suppliesthe information describing the selected constraint conditions as theconstraint condition information to the constraint condition settingunit 15061.

The constraint condition setting unit 15061 selects a constraintcondition expression corresponding to the constraint conditioninformation supplied from the user I/F 15016 from the constraintcondition expressions within the constraint condition setting unit 15061itself. Further, the constraint condition setting unit 15061 acquiresthe processing region width l based on the processing region informationof the input image supplied from the equation generating unit 15022 fromthe processing region setting unit 15011, and generates only the number,which corresponds to the processing region width l, of the constraintcondition expressions. For example, in the event that Expression (143)is used which has the condition “adjacent pixel difference=0” as theconstraint condition information, the constraint condition setting unit15061 generates the constraint condition expression expressed by theabove-described Expression (145), by acquiring the processing regionwidth l based on the processing region information supplied from theprocessing region setting unit 15011, and supplies this to the normalequation generating unit 15062. In the example of the processing region15042 illustrated in FIG. 119, the processing region 15042 isrectangular-shaped, and the processing region width l is the same ofeach line, but in the case that the processing region width l differsfor each line, the number of constraint condition expressions alsodiffer for each line.

The equation with k number of mixed pixel value P_(i), which is the sameas the number of lines, (Expression (141)) from the model generatingunit 15021 and the constraint condition Expression (Expression (145))corresponding to the constraint condition selected by the user operatingthe user I/F 15016 from the constraint condition setting unit 15061 aresupplied to the normal equation generating unit 15062.

First, as illustrated in FIG. 128, the normal equation generating unit15062 selects a predetermined line from the processing region 15042 as aline of interest. Then the normal equation generating unit 15062generates a normal equation corresponding to the selected line ofinterest, which is expressed by Expression (148) (Expression (149)),from a mixed pixel value P_(i) equation expressed by Expression (141)which is supplied from the model generating unit 15021, and a constraintcondition expression expressed by Expression (145) which is suppliedfrom the constraint condition setting unit 15061.

Further, the normal equation generating unit 15062 computes the matrix(A^(T)A)⁻¹A^(T), which is a coefficient of the column vector y of theright side of Expression (152), and substitutes the mixed pixel valueP_(i) which is acquired based on the input image supplied from thesensor 2 (FIG. 1) into the column vector y. The matrix (A^(T)A)⁻¹A^(T)is computed, and Expression (152) into which the mixed pixel value P_(i)is substituted is further supplied from the normal equation generatingunit 15062 to the actual world waveform estimating unit 15023.

When the normal equation generating unit 15062 finishes theabove-described generation of the normal equation, the computation ofthe matrix (A^(T)A)⁻¹A^(T), and the processing such as the substitutionof the mixed pixel value P_(i), a line not yet selected is selected asthe next line of interest, and similar processing is performed as to theselected next line of interest.

The actual world waveform estimating unit 15023, by computing Expression(152) which is supplied from the normal equation generating unit 15062,finds the column vector x, that is to say, the pixel value Q_(h) with nomovement blurring (approximation function), and supplies this to theimage generating unit 15014 (FIG. 113).

The image generating unit 15014 replaces a portion of the processingregion 15042 of the input image with the image of the pixel value Q_(h)with no movement blurring, which is supplied from the actual worldwaveform estimating unit 15023, based on the processing regioninformation supplied from the processing region setting unit 15011, andgenerates this as an output image, and supplies this to the imagedisplay unit 15015.

Next, the actual world estimating processing of the actual worldestimating unit 15013 in step S15008 in FIG. 114 will be described inndetail with reference to the flowchart in FIG. 130.

First, in step S15031, the model generating unit 15021 acquires theprocessing region width l and the number of lines k, based on theprocessing region information supplied from the processing regionsetting unit 15011, and the flow proceeds to step S15032.

In step S15032, the model generating unit 15021 acquires the movementamount v, which is equivalent to the number of lines k in the processingregion 15042, from the continuity setting unit 15012, and the flowproceeds to step S15033.

In step S15033, the model generating unit 15021 generates an equationfor the mixed pixel value P_(i) for each line, and the flow proceeds tostep S15034. In other words, the model generating unit 15021 generates knumber of model equations (expressions (141)).

In step S15034, the constraint condition setting unit 15061 selects theconstraint condition expression corresponding to the constraintcondition information supplied from the user I/F 15016, from theconstraint condition expressions existing within the constraintcondition setting unit itself. Also, the constraint condition settingunit 15061 acquires the processing region width l from the processingregion information of the input image which is supplied from theequation generating unit 15022 from the processing region setting unit15011, and generates an constraint condition Expression (Expression(145)) corresponding to the constraint condition information from theusers interface 15016, which comprises a number of equationscorresponding to the processing region width l, and the flow proceeds tostep S15035.

In step S15035, the normal equation generating unit 15062 selects onepredetermined line as the line of interest from the processing region15042, and the flow proceeds to step S15036.

In step S15036, the normal equation generating unit 15062 generates anormal equation expressed by Expression (148) (Expression (149)) fromthe equation (model equation) of the mixed pixel value P_(i) expressedby Expression (141) which is supplied from the model generating unit15021, and the equation (constraint condition expression) of theconstraint condition expressed by Expression (145) which is suppliedfrom the constraint condition setting unit 15061, and the flow proceedsto step S15037.

In step S15037, the normal equation generating unit 15062 computes thematrix (A^(T)A)⁻¹A^(T), which is a coefficient of the column vector y ofthe right side of Expression (152), and the flow proceeds to stepS15038.

In step S15038, the normal equation generating unit 15062 substitutesthe mixed pixel value P_(i) which is acquired based on the input imagesupplied from the sensor 2 (FIG. 1) into the column vector y, and theflow proceeds to step S15039.

In step S15039, the actual world waveform estimating unit 15023 computesa normal equation. That is to say, the actual world waveform estimatingunit 15023 find a pixel value Q_(h) with no movement blurring bycomputing Expression (152) which is supplied from the normal equationgenerating unit 15062, and the flow proceeds to step S15040.

In step S15040, determination is made as to whether or not processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042. In the event that it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15035, and the processing insteps S15035 through S15040 is repeated. That is to say, a line not yetselected of the processing region 15042 is selected as the next line ofinterest, and processing is performed to find a pixel value Q_(h) withno movement blurring.

On the other hand, in the event that it is determined that processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042, the actual world estimatingprocessing ends and returns.

Thus, by generating a normal equation (Expression (148)) as to one linewhich is the line of interest of the processing region 15042, andrepeating the processing to find a pixel value Q_(h) with no movementblurring k number of times, which is the number of lines in theprocessing region 15042, the pixel value Q_(h) with no movement blurringcan be found across the entire region of the processing region 15042.

In the event that the movement amount v and the processing region widthl are each the same for each line of the processing region 15042, thematrix A in Expression (148) is the same for every line. Accordingly,the matrix (A^(T)A)⁻¹A^(T) in Expression (148) also is the same forevery line, and the processing to compute the matrix (A^(T)A)⁻¹A^(T) foreach line can be omitted.

Next, the actual world estimating processing of the actual worldestimating unit 15013 in step S15008 in FIG. 114 in the event that themovement amount v and the processing region width l are each the samefor each line of the processing region 15042 will be described withreference to the flowchart of FIG. 131. However, the portions in FIG.131 which are similar to the flowchart in FIG. 130 will omit thedescription thereof as appropriate.

In FIG. 131, the processing in step S15057 which corresponds to stepS15035 in FIG. 130 is positioned not between step S15054 and S15055, butbetween steps S15056 and S15058. Further, in step S15060, if it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15057.

In other words, in the event that the movement amount v and theprocessing region width l of each line of the processing region 15042are each the same, the normal equation generated in step S15055 is thesame for each line, and therefore, after computing the matrix(A^(T)A)⁻¹A^(T) as to the first line of interest in step S15056,computing the matrix (A^(T)A)⁻¹A^(T) as to the remaining lines ofinterest (the processing in step S15056) is not necessary. Accordingly,the selection of the line of interest (step S15057), the substitution ofthe mixed pixel value P_(i) (the processing in step S15058), and thecomputation of the normal equation (the processing in step S15059) onlyneed be executed in order, as to the lines not selected in theprocessing region 15042.

Thus, with the embodiment illustrated in FIG. 113, a normal equation isgenerated from an equation of the mixed pixel value P_(i) wherein apixel value generating movement blurring is modeled with a pixel valuewith no movement blurring (model equation), and a constraint conditionexpression using the relation between adjacent pixels of a pixel valuewith no movement blurring, and by computing this normal equation, apixel value with no movement blurring (approximation function) is found,and therefore an image with good image quality can be generated.

FIGS. 132 through 134 are diagrams to describe the processing results bythe signal processing device 4 in FIG. 113.

FIG. 132 illustrates an input image to be input in the signal processingdevice 4 in FIG. 113. With the input image in FIG. 132, the characters“TAL” are moving in the direction from left to right at the same speed,and movement blurring is occurring in the horizontal direction.

FIG. 133 is an image using the above-described first method, that is tosay, a expression hypothesizing that the edge portion of the processingregion is flat is added to the model equation from the input image inFIG. 132 and a normal equation is generated, and the image in FIG. 133is generated by computing the normal equation. As is clear by comparingFIG. 132 and FIG. 133, in the image in FIG. 133, the movement blurringoccurring in the input image in FIG. 132 is reduced, and the characters“TAL” are relatively clearly displayed.

On the other hand, FIG. 134 is an image using the above-described secondmethod, that is to say, a normal equation is generated by adding aconditional expression constraining the relation of the adjacent pixelvalues from the input image in FIG. 132 to the model equation, and theimage in FIG. 134 is generated by computing the normal equation. In theimage in FIG. 134 also, the movement blurring occurring in the inputimage in FIG. 132 is reduced, and the characters “TAL” are clearlydisplayed.

With the embodiments in FIGS. 132 through 134, the image in FIG. 134 bythe second method is an image with has less noise and no movementblurring as compared to the image in FIG. 133 by the first method.

In the above example, the constraint condition information supplied fromthe user I/F 15016 to the constraint condition setting unit 15061 issaid to be information expressing which constraint condition the userhas selected (specified) from the several constraint conditions whichare set in advance internally in the constraint condition setting unit15061, but it can be such that the user can directly input theconstraint condition expression by operating the user I/F 15016.

With the embodiment of the signal processing device 4 illustrated inFIG. 113, an image with no movement blurring is generated such that thetotal sum E of the squared error expressed with Expression (150) becomesas small as possible, that is to say, generated such that the erroroccurring in the equation of the mixed pixel value P_(i) and the erroroccurring in the constraint condition expression becomes as small aspossible. Corresponding thereto, for example, of the error e_(mi)occurring in the equation of the mixed pixel value P_(i) of Expression(146) and the error e_(bj) occurring in the constraint conditionexpression of Expression (147), one error or the other can be made assmall as possible, and thus the balance (weighting) of the modelequation and the constraint condition expression can be adjusted andthus generate an image with no movement blurring.

FIG. 135 illustrates such a configuration example of the embodiment ofthe signal processing device 4, which generates an image with nomovement blurring by adjusting the weighting of the equation of themixed pixel value P_(i) (the model equation) and the equation of theconstraint conditions (constraint condition expression).

In other words, FIG. 135 illustrates a configuration example of anotherembodiment of the application example of the signal processing device 4illustrated in FIG. 111.

An image of one frame or one field, for example, as the data 3 of thesensor 2 is input into the signal processing device 4 in FIG. 135. Here,let us say that an object having a predetermined shape which moves at afixed speed of v pixels per shutter time (exposure time) in thehorizontal direction (sideways direction) of the image is displayed inthe input image. That is to say, the object is moving at the movementamount of v pixels in the horizontal direction, and therefore the lightsignal of the object and the light signal of the portions other than theobject are mixed (time mixture) by the time-integration effect of thesensor 2, and thus, the input image becomes a blurred image in portionssuch as the edge portions of the object. The signal processing device 4shown in FIG. 135 generates high-quality output images wherein movementblurring occurring due to such temporal mixing has been removed from theinput image.

In FIG. 135, the processing region setting unit 15081, the continuitysetting unit 15082, the actual world estimating unit 15083, imagegenerating unit 15084, image display unit 15085, and user I/F 15086,each correspond to the processing region setting unit 10001, thecontinuity setting unit 10002, the actual world estimating unit 10003,image generating unit 10004, image display unit 10005, and user I/F10006 shown in FIG. 111, and basically perform the same processing asthe processing region setting unit 10001, the continuity setting unit10002, the actual world estimating unit 10003, image generating unit10004, image display unit 10005, and user I/F 10006. Further, in FIG.135, the actual world estimating unit 15083 comprises a model generatingunit 15091, an equation generating unit 15092, and actual world waveformestimating unit 15093. The model generating unit 15091, equationgenerating unit 15092, and actual world waveform estimating unit 15093correspond to each of the model generating unit 10011, equationgenerating unit 10012, and actual world waveform estimating unit 10013in FIG. 111, and basically perform the same functions as each of themodel generating unit 10011, equation generating unit 10012, and actualworld waveform estimating unit 10013.

Note however, that in FIG. 135, the assisting information which the userI/F 15086 outputs by the user operating the user I/F 15086, is suppliedto the equation generating unit 15092.

In other words, in FIG. 135, by operating the user I/F 15086, with theequation generating unit 15092, the user can set the constraintconditions which constrain the relation between each of the pixels of animage with no movement blurring, which is an image of the actual world 1corresponding to the input image, and when the user performs anoperation to set the constraint conditions, the user I/F 15086 suppliesthe constraint condition information expressing the constraintconditions set by the operation to the equation generating unit 15092.

Specifically, for example, a user who views an image reflecting ablurring image displayed on the image display unit 15085 guesses theoriginal image, that is to say, an image without movement blurring whichis an image of the actual world 1 corresponding to the input image.Also, for example, the user guesses predetermined regions where blurringis occurring as the edge portions, and guesses the relation of the pixelvalues between each of the pixels (the constraint conditions), such asthe difference being large between the pixel values (level) between theadjacent pixels at the edge portions. Further, the user can set theconditions suitable for the guessed relation of the pixels value betweeneach of the pixels into the equation generating unit 15092, by operatingthe user I/F 15086.

Also, in FIG. 135, the processing region information, which is suppliedfrom the processing region setting unit 15081 to the actual worldestimating unit 15083, is supplied to the model generating unit 15091and the equation generating unit 15092 of the actual world estimatingunit 15083. The model generating unit 15091 and the equation generatingunit 15092 recognize the processing region in the input image from theprocessing region information which is supplied from the processingregion setting unit 15081.

The continuity setting unit 15082 sets a movement amount (movementvector) v in the input image as the continuity information that anobject is moving in the horizontal direction of the image at theconstant speed of v pixels per shutter time, and supplies this movementamount v as the continuity information to the model generating unit15091 of the actual world estimating unit 15083.

The model generating unit 15091 generates an equation (hereafterreferred to as model equation as appropriate) as a model (hereafterreferred to as relation model as appropriate), wherein the relationbetween the signal of the actual world 1 and the pixel values of each ofthe pixels of the movement blurring image as the input image whichconsiders the movement amount v as the continuity information suppliedfrom the continuity setting unit 15082, regarding the processing regionrecognized by the processing region information which is supplied fromthe processing region setting unit 15081, and supplies the generatedmodel equation to the equation generating unit 15092.

The equation generating unit 15092 generates a constraint conditionExpression (hereafter referred to as constraint condition expression asappropriate), based on the constraint condition information suppliedfrom the user I/F 15086. Also, the equation generating unit 15092 setsthe weighting into the constraint condition expression for adjusting thebalance of the model equation supplied from the model generating unit15091, and substitutes the pixel values of each of the pixels of theinput image comprising the processing region into the equation formedfrom the weighted constraint condition expression and the modelequation, and thus, generates an equation to find the approximationfunction as a model (hereafter referred to as approximation model)wherein the signal of the actual world 1 is modeled, and supplies thisto the actual world waveform estimating unit 15093.

The actual world waveform estimating unit 15093 estimates the waveformof the signal of the actual world 1 by computing the equation suppliedfrom the equation generating unit 15092. In other words, the actualworld waveform estimating unit 15093 finds the approximation function asan approximation model, and supplies the approximation function to theimage generating unit 15084 as the estimation result of the signalwaveform of the actual world 1. Note that approximation functionapproximating the actual world 1 signals include functions with constantfunction values, regardless of argument value.

The image generating unit 15084 recognizes the processing region in theinput image from the processing region information supplied from theprocessing regions setting unit 15081. Also, the image generating unit15084 generates a signal more closely approximating the signal of theactual world 1, from the approximation function supplied from the actualworld waveform estimating unit 15093, that is to say, generates an imagewith no movement blurring, and replaces the image of the processingregion of the input image with an output image wherein the image has nomovement blurring, generates this as an output image, and supplies thisto the image display unit 15085.

Next, the processing of the signal processing device 4 in FIG. 135 willbe described with reference to the flowchart in FIG. 136.

First, in step S15081, the signal processing device 4 performspre-processing, and the flow proceeds to step S15082. That is to say,the signal processing device 4 supplies one frame or one field, forexample, of the input image, supplied from the sensor 2 (FIG. 1) as data3, to the processing region setting unit 15081, continuity setting unit15082, actual world estimating unit 15083, image generating unit 15084,and image display unit 15085. Further, the signal processing unit 4causes the image display unit 15085 to display the input image.

In step S15082, the user I/F 15086 determines whether or not there hasbeen some sort of user input, by the user operating the user I/F 15086.In step S15082, in the event that determination is made that there is nouser input, i.e., in the event that the user has made no operations, theflow skips steps S15083 through S15085, and proceeds to step S15086.

On the other hand, in the event that determination is made in stepS15082 that there has been user input, i.e., in the event that the userhas viewed the input image displayed on the image display unit 15085 andoperated the user I/F 15086, thereby making user input indicating somesort of instruction or information, the flow proceeds to step S15083,where the user I/F 15086 determines whether or not the user input isending instructions for instructing ending of the processing of thesignal processing device 4.

In the event that determination is made in step S15083 that the userinput is ending instructions, the signal processing device 4 endsprocessing.

On the other hand, in the event that determination is made in stepS15083 that the user input is not ending instructions, the flow proceedsto step S15084, where the user I/F 15086 determines whether or not theuser input is constraint condition information. In the event thatdetermination is made in step S15084 that the user input is notconstraint condition information, the flow skips step S15085, andproceeds to step S15086.

Also, in step S15084, in the event that the user input is determined tobe constraint condition information, the flow proceeds to step S15085,and the user I/F 15086 supplies the constraint condition information tothe equation generating unit 15092, and the flow proceeds to stepS15086.

In step S15086, the processing region setting unit 15081 sets theprocessing region based on the input image, and supplies the processingregion information identifying the processing region to the continuitysetting unit 15082, model generating unit 15091 and equation generatingunit 15092 of actual world estimating unit 15083, and image generatingunit 15084, and the flow proceeds to step S15087. Here, in the settingof processing region in step S15086 the user inputs the processingregion instruction information by operating the user I/F 15086, and thiscan be performed based on the processing region instruction information,or it can be performed without processing region instruction informationfrom the user.

In step S15087, the continuity setting unit 15082 recognizes theprocessing region in the input image, from the processing regioninformation supplied from the processing region setting unit 15081.Further, the continuity setting unit 15082 sets the continuity of theactual world 1 signals lost from one portion of the continuity in theimage data at that processing region, and supplies continuityinformation representing that continuity to the model generating unit15091 of the actual world estimating unit 15083 and the flow proceeds tostep S15088.

Here the continuity setting unit 15082 sets the movement amount v,expressing that the movement is in the horizontal direction at aconstant speed of v pixels per shutter time (exposure time), as thecontinuity information, and supplies this to the model generating unit15091 of the actual world estimating unit 15083. The continuity settingunit 15082 sets the movement amount showing only the movement size ascontinuity information, under the premise that the object is moving in ahorizontal direction, but a movement vector showing the size anddirection of the movement of the object can be set as continuityinformation as well. The setting of continuity in step S15087 can bearranged by the user inputting the continuity instruction information byoperating the user I/F 15086, and can be performed based on thiscontinuity instruction information, or can be performed with nocontinuity instruction information from the user.

The actual world estimating unit 15083 performs actual world estimatingprocessing in step S15008. In other words, with the actual worldestimating unit 15083, the model generating unit 15091 generates anequation (model equation) as the model (relation model) wherein therelation of the pixel values of each of the pixels of the input imagewhich has movement blurring and the signal of the actual world 1 ismodeled, based on the movement amount v supplied from the continuitysetting unit 15082 in step S15087 and the processing region informationsupplied from the processing region setting unit 15081 in step S15086,and the generated model equation is supplied to the equation generatingunit 15092.

The equation generating unit 15092 generates a constraint conditionexpression based on the constraint condition information supplied fromthe user I/F 15086 in step S15085, and sets the weighting for adjustingthe balance with the model equation in the generated constraintcondition expression. Also, the equation generating unit 15092 generatesan equation for finding the approximation function as the model(approximation model) wherein the signal of the actual world 1 ismodeled, from the weighted constraint condition expression and the modelequation supplied from the model generating unit 15091 to the equationgenerating unit 15092, and substitutes the pixel values of each of thepixels of the input image into this equation, and supplies this to theactual world waveform estimating unit 15093. In the event that the userhas not specified (set) constraint condition information, that is tosay, in the event that NO is determined to be the case in step S15082 orS15084, the equation generating unit 15092 performs predeterminedprocessing such as a method for solving a simultaneous equation formedof a model equation and an equation hypothesizing that the edge portionsof the processing region is “flat”, as described in the embodiment inFIG. 113, for example.

Now, the weighting can be set for the constraint condition expressiononly, for the model equation only, or for both the constraint conditionexpression and the model equation.

The actual world waveform estimating unit 15093 estimates the waveformof the actual world 1 signal, that is to say, finds the approximationfunction as a model wherein the actual world 1 signal is modeled, bycomputing the equation supplied from the equation generating unit 15092,and the approximation function is supplied to the image generating unit15084 as the estimation result of the signal waveform of the actualworld 1.

The details of the actual world estimating process of step S15088 willbe described later with reference to FIG. 138.

After the processing in step S15088, the flow proceeds to step S15089,and the image generating unit 15084 generates a signal more closelyapproximating the signal of the actual world 1, from the approximationfunction supplied from the actual world waveform estimating unit 15093,that is to say, generates an image with no movement blurring, and basedon the processing region information supplied from the processing regionsetting unit 15081, converts the portion of the processing region of theinput image into an image which has no movement blurring generated,generates this as an output image, and supplies this to the imagedisplay unit 15085, and the flow proceeds to step S15090.

In step S15090, the image display unit 15085 displays the output imagesupplied from the image generating unit 15084 instead of the input imagedisplayed in step S15081, or along with the input image, and the flowproceeds to step S15091.

In step S15091, as with the case of step S15082, the user I/F 15086determines whether or not there has been some sort of user input by theuser operating the user I/F 15086, and in the event that determinationis made that there has been no user input, i.e., in the event that theuser has made no operations, the flow returns to step S15091, and awaitssome sort of user input.

Also, in the event that determination is made in step S15091 that therehas been user input, i.e., in the event that the user has viewed theinput image or output image displayed on the image display unit 15085and operated the user I/F 15086, thereby making user input representingsome sort of instructions or information, the flow proceeds to stepS15092, where the user I/F 15086 determines whether or not the userinput is ending instructions instructing ending of the processing of thesignal processing device 4.

In the event that determination is made in step S15092 that the userinput is ending instructions, the signal processing device 4 endsprocessing.

On the other hand, in the event that determination is made in stepS15092 that the user input is not ending instructions, the flow proceedsto step S15093, where the user I/F 15086 determines whether or not theuser input is constraint condition information. In the event thatdetermination is made in step S15093 that the user input is notconstraint condition information, the flow returns to step S15091, andthe same processing is repeated thereafter.

Also, in step S15093, in the event that determination is made that theuser input is constraint condition information, the flow returns to stepS15085, and as described above, the user I/F 15086 supplies theconstraint condition information to equation generating unit 15092. Theflow then proceeds from step S15085 to step S15086, and hereafter thesame processing is repeated. Now, in the event that the signalprocessing device 4 repeats the processing of step S15085 through stepS15093, the processing of the above-described step S15086 and S15087 arethe same as that set by the first processing S15086 and S15087, and thesame process as the first process may be repeated, or may be skipped.

Thus, with the signal processing device 4 in FIG. 135, by finding anapproximation function of a model wherein the signal of the actual world1 is modeled, an output image of high image quality with no movementblurring can be generated.

Also, by operating the user I/F 15086, the user can set constraintconditions which constrain the relation between each of the pixels inthe original image having no movement blurring, and in the event that anew constraint condition is input, an image without movement blurringcan be obtained again, and therefore an output image of high imagequality according to the preference of the user can be easily acquired.

In the signal processing device 4 in FIG. 135, an output image withoutmovement blurring is generated by finding an approximation functionwhich approximates the light signal of the actual world 1, but thisoutput image can be regarded as having the movement blurring removedfrom the input image. Accordingly, it can be said that processing isperformed for movement blurring removal with the signal processingdevice 4 in FIG. 135.

Next, the internal configuration of the actual world estimating unit15083 in FIG. 135 will be described with reference to FIG. 137. With theembodiment in FIG. 135 also, the mechanism for generating movementblurring of objects reflecting in the input image is modeled as arelation of the above-described Expression (141), which is an exampleshown in FIGS. 125 and 126.

In other words, the pixel values of each of the pixels of the movementblurring image as the input image are values wherein the pixel values(or the electric charge corresponding thereto) of each of the pixels ofan image with no movement blurring which accumulate (integrate) whilemoving at the movement amount v, and models the mechanism for movementblurring generating.

The actual world estimating unit 15083 comprises a model generating unit15091, an equation generating unit 15092, and an actual world waveformestimating unit 15093. Further, the equation generating unit 15092comprises a constraint condition setting unit 15101, a weightingchanging unit 15012, and a normal equation generating unit 15103.

The movement amount v as the continuity information is supplied to themodel generating unit 15091 by the continuity setting unit 15082. Themovement amount v in this example expresses the continuity of the objectmoving in the horizontal direction at a constant speed of v pixels pershutter time, within the processing region of the input image which isset in the processing region setting unit 15081, and for example, v=4,as with the above-described example. Also, the movement amount v may bea value which differs for each line, and in the event that the movementamount v differs for each line, k number, equal to the number of lines,of the movement amount v is supplied as continuity information from thecontinuity setting unit 15082 to the model generating unit 15091.

Also, processing region information which specifies the processingregion within the input image is also supplied to the model generatingunit 15091 from the processing region setting unit 15081. In the case ofusing a rectangular-shaped region as the processing region, theprocessing region information can be expressed, for example, bycoordinate values at opposite angles of a rectangular-shaped processingregion as to an absolute coordinates system wherein a predeterminedposition (pixel) of the input image is the origin, or the centercoordinate of the processing region and the width (horizontal direction)and the height (vertical direction) of the processing region.

The model generating unit 15091 acquires, for example, a processingregion width of l, and k line numbers, as illustrated in FIG. 119, fromthe processing region information supplied from the processing regionsetting unit 15081. In this example, let us say that the processingregion 15042 is rectangular-shaped, and for example, has l=10 and k=4.In this case, the processing region width of each line (the number ofpixels) are the same, but with the processing region 15042, each linemay have a different processing region width. In the case that theprocessing region width l differs for each line, the model generatingunit 15091 acquires a processing region width l of the same number (k)as the number of lines.

Also, the model generating unit 15091 generates an equation (modelequation) of a mixed pixel value P_(i) which is shown in theabove-described Expression (141), based on the movement amount vsupplied from the continuity setting unit 15012 and the processingregion width l and k number of lines acquired from the processing regioninformation, and supplies this to the normal equation generating unit15103.

The constraint condition information specified by the user operating theuser I/F 15086 is supplied to the constraint condition setting unit15101 from the user I/F 15086. For example, the user performs anoperation with the user I/F 15086 to select (specify) the desiredconstraint condition from the constraint conditions which constrain therelation of the pixel values Q_(h) of each of the pixels of an imagewith no movement blurring, such as the above-described Expression (143)or Expression (144), which have been set in advance inside theconstraint condition setting unit 15101, and the user I/F 15086 suppliesthe information describing the selected constraint conditions as theconstraint condition information to the constraint condition settingunit 15101.

The constraint condition setting unit 15101 selects a constraintcondition expression corresponding to the constraint conditioninformation supplied from the user I/F 15086 from the constraintcondition expressions within the constraint condition setting unit 15101itself. Further, the constraint condition setting unit 15101 acquiresthe processing region width l based on the processing region informationof the input image supplied from the equation generating unit 15092 fromthe processing region setting unit 15081, and generates only the numberwhich corresponds to the processing region width l, for the constraintcondition expressions. For example, in the event that Expression (143)is used which has the condition “adjacent pixel difference=0” as theconstraint condition information, the constraint condition setting unit15101 generates the constraint condition expression expressed by theabove-described Expression (145), by acquiring the processing regionwidth l based on the processing region information supplied from theprocessing region setting unit 15081, and supplies this to the normalequation generating unit 15103. In the example of the processing region15042 illustrated in FIG. 119, the processing region 15042 isrectangular-shaped, and the processing region width l is the same ofeach line, but in the case that the processing region width l differsfor each line, the number of constraint condition expressions alsodiffer for each line.

A weighting W_(m) as to the model equation and a weighting W_(b) as tothe constraint condition expression are set in advance in the weightingchanging unit 15102. With this embodiment, the weighting coefficient Wrelating to the constraint condition expression is set by Expression(153), and using only the weighting relating to the constraint conditionexpression, the balance of the model equation and the constraintcondition expression is adjusted.

$\begin{matrix}{W = {W_{b}/W_{m}}} & (153)\end{matrix}$

The weighting changing unit 15102 changes the weighting of the modelequation and the constraint condition expression in the normal equationgenerated by the normal equation generating unit 15103, by supplying theweighting coefficient W to the normal equation generating unit 15103.

The mixed pixel value P_(i) equation with the same number of k as theline numbers from the model generating unit 15091 (Expression (141)) andthe constraint condition Expression (Expression (145)) corresponding tothe constraint conditions selected from the constraint condition settingunit 15101 by the user operating the user I/F 15086 are supplied to thenormal equation generating unit 15103.

With the normal equation generating unit 15103, the weightingcoefficient W is set by the weighting changing unit 15102, regardingExpression (147) wherein the error e_(bj) is considered in theconstraint condition Expression (145) which is supplied from theconstraint condition setting unit 15101 (the weighting W as to the modelequation and constraint condition [equation?] which was 1:1 is changed).

In other words, Expression (147) is changed into the followingExpression (154) of which both sides are multiplied by the weighting W.Here, the weighting coefficient W is for example a predetermined valuesuch as 0.1. In this case, a pixel value Q_(h), which makes the errore_(mi) of the model equation smaller than the error e_(bj) of theconstraint condition expression, is found.

$\begin{matrix}\begin{matrix}{{W\left( {Q_{0} - Q_{1}} \right)} = {We}_{b\; 0}} \\{{W\left( {Q_{1} - Q_{2}} \right)} = {We}_{b\; 1}} \\{{W\left( {Q_{2} - Q_{3}} \right)} = {We}_{b\; 2}} \\{{W\left( {Q_{3} - Q_{4}} \right)} = {We}_{b\; 3}} \\{{W\left( {Q_{4} - Q_{5}} \right)} = {We}_{b\; 4}} \\{{W\left( {Q_{5} - Q_{6}} \right)} = {We}_{b\; 5}} \\{{W\left( {Q_{6} - Q_{7}} \right)} = {We}_{b\; 6}} \\{{W\left( {Q_{7} - Q_{8}} \right)} = {We}_{b\; 7}} \\{{W\left( {Q_{8} - Q_{9}} \right)} = {We}_{b\; 8}}\end{matrix} & (154)\end{matrix}$

The normal equation generating unit 15103 selects one predetermined lineas the line of interest from the processing region 15042 as shown inFIG. 128. Then, the normal equation generating unit 15103 generates thenormal equation expressed by the following Expression (155), from themixed pixel value P_(i) equation (model equation) expressed byExpression (141) which is supplied from the model generating unit 15091corresponding to the selected line of interest, and the constraintcondition expression expressed by Expression (154) wherein the weightingcoefficient W is added to Expression (145) which is supplied from theconstraint condition setting unit 15101.

$\begin{matrix}{\begin{bmatrix}{1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} \\W & {- W} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & W & {- W} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & W & {- W} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & W & {- W} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & W & {- W} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & W & {- W} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & W & {- W} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & W & {- W} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & W & {- W}\end{bmatrix}{\quad{\quad\mspace{461mu}{\begin{bmatrix}Q_{0} \\Q_{1} \\Q_{2} \\Q_{3} \\Q_{4} \\Q_{5} \\Q_{6} \\Q_{7} \\Q_{8} \\Q_{9}\end{bmatrix} = {\begin{bmatrix}P_{3} \\P_{4} \\P_{5} \\P_{6} \\P_{7} \\P_{8} \\P_{9} \\0 \\0 \\0 \\0 \\0 \\0 \\0 \\0 \\0\end{bmatrix} + \begin{bmatrix}e_{m\; 3} \\e_{m\; 4} \\e_{m\; 5} \\e_{m\; 6} \\e_{m\; 7} \\e_{m\; 8} \\\underset{\_}{e_{m\; 9}} \\{W\; e_{b\; 0}} \\{W\; e_{b\; 1}} \\{W\; e_{b\; 2}} \\{W\; e_{b\; 3}} \\{W\; e_{b\; 4}} \\{W\; e_{b\; 5}} \\{W\; e_{b\; 6}} \\{W\; e_{b\; 7}} \\{W\; e_{b\; 8}}\end{bmatrix}}}}}} & (155)\end{matrix}$

e_(mi) is the error occurring in each expression in Expression (141),and e_(bj) is the error occurring in each expression in Expression(154).

Here, Expression (155) will be replaced by Expression (156), using thematrix A′ and the column vectors x, y, and e′.

$\begin{matrix}{{A^{\prime}x} = {y + e^{\prime}}} & (156)\end{matrix}$

In this case, the total sum E′ of the squared error of the error e_(mi)of Expression (146) and the error e_(bj) of Expression (154) can beexpressed with the following expression.

$\begin{matrix}{E^{\prime} = {{\sum e_{mi}^{2}} + {W^{2}{\sum e_{bj}^{2}}}}} & (157)\end{matrix}$

In order to find the Q_(h) for which the total sum E of the squarederror is the smallest, the column vector x (the column vector x with thepixel value Q_(h) as a component) can be expressed by Expression (158),from the equation similar to the above-described equations (150) and(151).

$\begin{matrix}{x = {\left( {A^{\prime\; T}A^{\prime}} \right)^{- 1}A^{\prime\; T}y}} & (158)\end{matrix}$

Now, with Expression (158), the superscript T means transposing, and thesuperscript −1 means an inverse matrix.

With Expression (157), when the weighting coefficient W is small, asharp image can be found which can better satisfy Expression (141) evenwith some noise allowed, and on the other hand, when the weightingcoefficient W is large, an image which is flat with little noise thatcan satisfy Expression (145), rather than Expression (141), can befound. That is to say, by adjusting the value of the weightingcoefficient W, the balance of “sharpness” and “noise” can be adjusted.

The normal equation generating unit 15103 computes the matrix(A′^(T)A′)⁻¹A′^(T), which is a coefficient of the column vector y of theright side of Expression (158), and substitutes the mixed pixel valueP_(i) which is acquired based on the input image supplied from thesensor 2 (FIG. 1) into the column vector y. The matrix(A′^(T)A′)⁻¹A′^(T) is computed, and Expression (158) into which themixed pixel value Pi is substituted is further supplied from the normalequation generating unit 15103 to the actual world waveform estimatingunit 15093.

When the normal equation generating unit 15103 finishes theabove-described generation of the normal equation, the computation ofthe matrix (A′^(T)A′)⁻¹A′^(T), and the processing such as thesubstitution of the mixed pixel value P_(i) regarding the selected lineof interest, a line not yet selected is selected as the next line ofinterest, and similar processing is performed as to the selected nextline of interest.

The actual world waveform estimating unit 15093, by computing Expression(158) which is supplied from the normal equation generating unit 15103,finds the column vector x, that is to say, the pixel value Q_(h) with nomovement blurring, and supplies this to the image generating unit 15084(FIG. 135).

The image generating unit 15084 replaces a portion of the processingregion 15042 of the input image with the image of the pixel value Q_(h)with no movement blurring, which is supplied from the actual worldwaveform estimating unit 15093, based on the processing regioninformation supplied from the processing region setting unit 15081, andgenerates this as an output image, and supplies this to the imagedisplay unit 15085.

Next, the actual world estimating processing of the actual worldestimating unit 15083 in step S15088 in FIG. 136 will be described withreference to the flowchart in FIG. 138.

First, in step S15101, the model generating unit 15091 acquires theprocessing region width l and the number of lines k, based on theprocessing region information supplied from the processing regionsetting unit 15081, and the flow proceeds to step S15102.

In step S15102, the model generating unit 15091 acquires the movementamount v, which is equivalent to the number of lines k in the processingregion 15042, from the continuity setting unit 15082, and the flowproceeds to step S15103.

In step S15103, the model generating unit 15091 generates an equationfor the mixed pixel value P_(i) for each line, and the flow proceeds tostep S15104. In other words, the model generating unit 15091 generates knumber of model equations (expressions (141)).

In step S15104, the constraint condition setting unit 15101 selects theconstraint condition expression corresponding to the constraintcondition information supplied from the user I/F 15086, from theconstraint condition expressions existing within the constraintcondition setting unit itself. Also, the constraint condition settingunit 15101 acquires the processing region width l from the processingregion information of the input image which is supplied from theequation generating unit 15092 from the processing region setting unit15081, and generates an constraint condition Expression (Expression(145)) corresponding to the constraint condition information from theuser I/F 15086, which comprises a number of equations corresponding tothe processing region width l, and the flow proceeds to step S15105.

In step S15105, the weighting changing unit 15102 supplies the weightingcoefficient W expressed by Expression (153), which is set in advanceinternally within the weighting changing unit 15102 itself, to thenormal equation generating unit 15103, and changes the weighting as tothe constraint condition expression, and the flow proceeds to stepS15106. That is to say, the weighting changing unit 15102 generates theabove-described Expression (154) at the normal equation generating unit15103.

In step S15106, the normal equation generating unit 15103 selects onepredetermined line as the line of interest from the processing region15042, and the flow proceeds to step S15107.

In step S15107, the normal equation generating unit 15103 generates anormal equation expressed by Expression (155) (Expression (156)) fromExpression (146), wherein the errore_(mi) is considered of the variousexpressions of the equation of the mixed pixel value P_(i) expressed byExpression (141), which is supplied from the model generating unit15091, and the constraint condition Expression (154) obtained bychanging the weighting of Expression (147) wherein the error e_(bj) isconsidered regarding Expression (145) which is supplied from theconstraint condition setting unit 15101, by the weighting changing unit15102 in step S15105, and the flow proceeds to step S15108.

In step S15108, the normal equation generating unit 15103 computes thematrix (A′^(T)A′)⁻¹A′^(T), which is a coefficient of the column vector yof the right side of Expression (158), and the flow proceeds to stepS15109.

In step S15109, the normal equation generating unit 15103 substitutesthe mixed pixel value P_(i) which is acquired based on the input imagesupplied from the sensor 2 (FIG. 1) into the column vector y, and theflow proceeds to step S15110.

In step S15110, the actual world waveform estimating unit 15093 computesa normal equation. That is to say, the actual world waveform estimatingunit 15093 finds a pixel value Q_(h) with no movement blurring bycomputing Expression (158) which is supplied from the normal equationgenerating unit 15103, and the flow proceeds to step S15111.

In step S15111, determination is made as to whether or not processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042. In the event that it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15106, and the processing insteps S15106 through S15111 is repeated. That is to say, a line not yetselected of the processing region 15042 is selected as the next line ofinterest, and processing is performed to find a pixel value Q_(h) withno movement blurring.

On the other hand, in the event that it is determined that processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042, the actual world estimatingprocessing ends and returns.

Thus, by generating a normal equation (Expression (155)) as to one linewhich is the line of interest of the processing region 15042, andrepeating the processing to find a pixel value Q_(h) with no movementblurring k number of times, which is the number of lines in theprocessing region 15042, the pixel value Q_(h) with no movement blurringcan be found across the entire region of the processing region 15042.

In the event that the movement amount v and the processing region widthl are each the same for each line of the processing region 15042, thematrix A in Expression (155) is the same for every line. Accordingly,the matrix (A′^(T)A′)⁻¹A′^(T) in Expression (158) also is the same forevery line, and the processing to compute the matrix (A′^(T)A′)⁻¹A′^(T)for each line can be omitted.

Next, the actual world estimating processing of the actual worldestimating unit 15083 in step S15088 in FIG. 136 in the event that themovement amount v and the processing region width l are each the samefor each line of the processing region 15042 will be described withreference to the flowchart of FIG. 139. However, the portions in FIG.139 which are similar to the flowchart in FIG. 138 will omit thedescription thereof as appropriate.

In FIG. 139, the processing in step S15138 which corresponds to stepS15106 in FIG. 138 is positioned not between step S15135 and S15136, butbetween steps S15137 and S15139. Further, in step S15141, if it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15138.

In other words, in the event that the movement amount v and theprocessing region width l of each line of the processing region 15042are each the same, the normal equation generated in step S15136 is thesame for each line, and therefore, after computing the matrix(A′^(T)A′)⁻¹A′^(T) as to the first line of interest in step S15137,computing the matrix (A′^(T)A′)⁻¹A′^(T) as to the remaining lines ofinterest (the processing in step S15137) is not necessary. Accordingly,the selection of the line of interest (step S15138), the substitution ofthe mixed pixel value P_(i) (the processing in step S15139), and thecomputation of the normal equation (the processing in step S15140) onlyneed be executed in order, as to the lines not selected in theprocessing region 15042.

Thus, with the embodiment illustrated in FIG. 135, a normal equation isgenerated from an equation of the mixed pixel value P_(i) wherein apixel value generating movement blurring is modeled with a pixel valuewith no movement blurring (model equation), and a constraint conditionexpression using the relation between adjacent pixels of a pixel valuewith no movement blurring, and by computing this normal equation, apixel value with no movement blurring (approximation function) is found,and therefore an image with good image quality can be generated.

Also, because the balance (weighting) of the mixed pixel value Piequation and the constraint condition expression can be adjusted, animage can be generated with no movement blurring, which considers“sharpness” and “noise” and has the image quality according to thepreference of the user.

With the above-described example, the constraint condition informationsupplied from the user I/F 15086 to the constraint condition settingunit 15101 is said to be information showing which constraint conditionthe user has selected (instructed), out of the multiple constraintconditions previously set internally in the constraint condition settingunit 15101, but it may be set up such that that user can directly inputthe constraint condition expression by operating the user I/F 15086.

With the embodiment of the signal processing device 4 in FIG. 135, theweighting coefficient W relating to the constraint condition expression,which adjusts the weighting (balance) of the mixed pixel value Piequation and the constraint condition expression, has been apredetermined value which is set in advance (0.1 in the above case).

However, setting the weighting coefficient W relating to the constraintcondition expression as a predetermined value (fixed value) is to applythe constraint conditions as to the places (relation between pixels)wherein the constraint condition is not established, similar to theplaces wherein the constraint condition is established, which causesdeterioration of the processing image in some cases.

Therefore, next, an embodiment will be described wherein the signalprocessing device 4 can determine the weighting relating to theconstraint condition expression and so forth corresponding to theactivity (features) of the input image.

FIG. 140 is a configuration example of another embodiment with anapplication example of the signal processing device 4 illustrated inFIG. 111.

An image of, for example, one frame or one field as the data 3 from thesensor 2 is input into the signal processing device 4 in FIG. 140. Here,an object having a predetermined shape is displayed in the input image,such object moving in the horizontal direction (crosswise direction) ofthe image at a constant speed of v pixels per shutter time (exposuretime). In other words, regarding the input image, the object is movingin a horizontal direction at the movement amount of v pixels, andtherefore because of the time integration effect of the sensor 2, thelight signal of the object and the light signal of the portion otherthan the object are mixed (time mixture), and thus, the image is blurredat portions such as the borders of the object. With the signalprocessing device 4 in FIG. 140, a high-quality output image isgenerated which has removed movement blurring from the input image whathad occurred due to such time mixture.

In FIG. 140, the processing region setting unit 15111, the continuitysetting unit 15112, the actual world estimating unit 15113, imagegenerating unit 15114, image display unit 15115, and user I/F 15116,each correspond to the processing region setting unit 10001, thecontinuity setting unit 10002, the actual world estimating unit 10003,image generating unit 10004, image display unit 10005, and user I/F10006 shown in FIG. 111, and basically perform the same processing asthe processing region setting unit 10001, the continuity setting unit10002, the actual world estimating unit 10003, image generating unit10004, image display unit 10005, and user I/F 10006. Further, in FIG.140, the actual world estimating unit 15113 comprises a model generatingunit 15121, an equation generating unit 15122, and actual world waveformestimating unit 15123. The model generating unit 15121, equationgenerating unit 15122, and actual world waveform estimating unit 15123correspond to each of the model generating unit 10011, equationgenerating unit 10012, and actual world waveform estimating unit 10013in FIG. 111, and basically perform the same functions as each of themodel generating unit 10011, equation generating unit 10012, and actualworld waveform estimating unit 10013.

Note however, that in FIG. 140, the assisting information which the userI/F 15116 outputs by the user operating the user I/F 15116, is suppliedto the equation generating unit 15122.

In other words, in FIG. 140, by operating the user I/F 15116, with theequation generating unit 15122, the user can set the constraintconditions which constrain the relation between each of the pixels of animage with no movement blurring, which is an image of the actual world 1corresponding to the input image, and when the user performs anoperation to set the constraint conditions, the user I/F 15116 suppliesthe constraint condition information expressing the constraintconditions set by the operation to the equation generating unit 15122.

Specifically, for example, a user who views an image displayed on theimage display unit 15115 reflecting a blurring image guesses theoriginal image, that is to say, an image without movement blurring whichis an image of the actual world 1 corresponding to the input image.Also, for example, the user guesses predetermined regions where blurringis occurring as the edge portions, and guesses the relation of the pixelvalues between each of the pixels (the constraint conditions), such asthe difference being large between the pixel values (level) between theadjacent pixels at the edge portions. Further, the user can set theconditions suitable for the guessed relation of the pixels value betweeneach of the pixels into the equation generating unit 15122, by operatingthe user I/F 15116.

Also, in FIG. 140, the processing region information, which is suppliedfrom the processing region setting unit 15111 to the actual worldestimating unit 15113, is supplied to the model generating unit 15121and the equation generating unit 15122 of the actual world estimatingunit 15113. The model generating unit 15121 and the equation generatingunit 15122 recognize the processing region in the input image from theprocessing region information supplied from the processing regionsetting unit 15111.

The continuity setting unit 15112 sets a movement amount (movementvector) v in the input image as the continuity information that anobject is moving in the horizontal direction of the image at theconstant speed of v pixels per shutter time, and supplies this movementamount v as the continuity information to the model generating unit15121 of the actual world estimating unit 15113.

The model generating unit 15121 generates an equation (hereafterreferred to as model equation as appropriate) as a model (hereafterreferred to as relation model as appropriate), modeling the relationbetween the signal of the actual world 1 and the pixel values of each ofthe pixels of the movement blurring image as the input image whichconsiders the movement amount v as the continuity information suppliedfrom the continuity setting unit 15112, regarding the processing regionrecognized by the processing region information which is supplied fromthe processing region setting unit 15111, and supplies the generatedmodel equation to the equation generating unit 15122.

The equation generating unit 15122 generates a constraint conditionExpression (hereafter referred to as constraint condition expression asappropriate), based on the constraint condition information suppliedfrom the user I/F 15116. Also, the equation generating unit 15122 setsthe weighting to be determined by the activity of the input image intothe constraint condition expression and the model equation supplied fromthe model generating unit 15121, and substitutes the pixel values ofeach of the pixels of the input image comprising the processing regioninto the equation formed from the weighted constraint conditionexpression and the model equation, and thus, generates an equation tofind the approximation function as a model (hereafter referred to asapproximation model) wherein the signal of the actual world 1 ismodeled, and supplies this to the actual world waveform estimating unit15123.

The actual world waveform estimating unit 15123 estimates the waveformof the actual world 1 signals, by computing the equation supplied fromthe equation generating unit 15122. In other words, the actual worldwaveform estimating unit 15123 finds the approximation function as anapproximation model, and supplies the approximation function to theimage generating unit 15114 as the estimation result of the signalwaveform of the actual world 1. Note that approximation functionapproximating the actual world 1 signals include functions with constantfunction values, regardless of argument value.

The image generating unit 15114 recognizes the processing region in theinput image from the processing region information supplied from theprocessing region setting unit 15111. Also, the image generating unit15114 generates a signal more closely approximating the signal of theactual world 1, from the approximation function supplied from the actualworld waveform estimating unit 15123, that is to say, generates an imagewith no movement blurring, and replaces the image of the processingregion of the input image with an image wherein the image has nomovement blurring, generates this as an output image, and supplies thisto the image display unit 15115.

Next, the processing of the signal processing device 4 in FIG. 140 willbe described with reference to the flowchart in FIG. 141.

First, in step S15161, the signal processing device 4 performspre-processing, and the flow proceeds to step S15162. That is to say,the signal processing device 4 supplies one frame or one field, forexample, of the input image, supplied from the sensor 2 (FIG. 1) as data3, to the processing region setting unit 15111, continuity setting unit15112, actual world estimating unit 15113, image generating unit 15114,and image display unit 15115. Further, the signal processing unit 4causes the image display unit 15115 to display the input image.

In step S15162, the user I/F 15116 determines whether or not there hasbeen some sort of user input, by the user operating the user I/F 15116.In step S15162, in the event that determination is made that there is nouser input, i.e., in the event that the user has made no operations, theflow skips steps S15163 through S15165, and proceeds to step S15166.

On the other hand, in step S15162, in the event that determination ismade that there has been user input, i.e., in the event that the userhas viewed the input image displayed on the image display unit 15115 andoperated the user I/F 15116, thereby making user input indicating somesort of instruction or information, the flow proceeds to step S15163,where the user I/F 15116 determines whether or not the user input isuser instructions instructing ending of the processing of the signalprocessing device 4.

In the event that determination is made that in step S15163 that theuser input is ending instructions, the signal processing device 4 endsthe processing.

On the other hand, in the event that determination is made in stepS15163 that the user input is not ending instructions, the flow proceedsto step S15164, where the user I/F 15166 determines whether or not theuser input is constraint condition instruction information. In the eventthat determination is made in step S15164 that the user input is notconstraint condition instruction information, step S15165 is skipped,and the flow proceeds to step S15166.

Also, in step S15164, in the event that the user input is determined tobe constraint condition information, the flow proceeds to step S15165,and the user I/F 15116 supplies the constraint condition information tothe equation generating unit 15122, and the flow proceeds to stepS15166.

In step S15166, the processing region setting unit 15111 sets theprocessing region based on the input image, and supplies the processingregion information identifying the processing region to the continuitysetting unit 15112, the model generating unit 15121 and equationgenerating unit 15122 of the actual world estimating unit 15113, andimage generating unit 15114, and the flow proceeds to step S15167. Now,an arrangement maybe made wherein the setting of the processing regionin step S15166 is performed by the user inputting processing regioninstruction information by operating the user I/F 15116, so as to becarried out based on the processing region instruction information, or,this may be carried out without processing region instructioninformation from the user.

In step S15167, the continuity setting unit 15112 recognizes theprocessing region in the input image, from the processing regioninformation supplied from the processing region setting unit 15111.Further, the continuity setting unit 15112 sets continuity of a part ofthe continuity of the actual world 1 signals that has been lost in theimage data of the processing region, and supplies continuity informationindicating the continuity thereof to the model generating unit 15121 ofthe actual world estimating unit 15113, and the flow proceeds to stepS15168. Here, the continuity setting unit 15112 sets the movement amountv, expressing that the movement is in the horizontal direction at aconstant speed of v pixels per shutter time (exposure time), as thecontinuity information, and supplies this to the model generating unit15121 of the actual world estimating unit 15113. Here, the continuitysetting unit 15112 sets the movement amount showing only the movementsize as continuity information, under the premise that the object ismoving in a horizontal direction, but a movement vector showing the sizeand direction of the movement of the object can be set as continuityinformation as well. Now, an arrangement may be made wherein the settingof the continuity in step S15167 is performed by the user inputtingcontinuity instruction information by operating the user I/F 15116, soas to be carried out based on the continuity instruction information,or, this may be carried out without continuity instruction information

The actual world estimating unit 15113 performs actual world estimatingprocessing in step S15168. In other words, with the actual worldestimating unit 15113, the model generating unit 15121 generates anequation (model equation) as the model (relation model) wherein therelation of the pixel values of each of the pixels of the input imagewhich has movement blurring and the signal of the actual world 1 ismodeled, based on the movement amount v supplied from the continuitysetting unit 15112 in step S15167 and the processing region informationsupplied from the processing region setting unit 15111 in step S15166,and the generated model equation is supplied to the equation generatingunit 15122.

The equation generating unit 15122 generates a constraint conditionexpression based on the constraint condition information supplied fromthe user I/F 15116 in step S15165, and sets the weighting determinedfrom the activity of the input image into the generated constraintcondition expression and the model equation supplied from the modelgenerating unit 15121 to the equation generating unit 15122. Then, theequation generating unit 15122 substitutes the pixel values of each ofthe pixels of the input image comprising the processing region into theequation formed from the weighted constraint condition expression andthe model equation, and thus generates the equation for finding theapproximation function as a model (approximate model) wherein the signalof the actual world 1 is modeled, and supplies this to the actual worldwaveform estimating unit 15123. In the event that the user has notspecified (set) constraint condition information, that is to say, in theevent that NO is determined to be the case in step S15162 or S15164, theequation generating unit 15122 performs predetermined processing such asa method for solving a simultaneous equation formed of a model equationand an equation hypothesizing that the edge portions of the processingregion is “flat”, as described in the embodiment in FIG. 113, forexample.

The actual world waveform estimating unit 15123 estimates the waveformof the actual world 1 signals, by computing the equation supplied fromthe equation generating unit 15122, that is to say, the actual worldwaveform estimating unit 15123 obtains an approximation functionmodeling the actual world 1 signals, and supplies the approximationfunction to the image generating unit 15114, as estimation results ofthe waveform of the actual world 1 signals.

The details of the actual world estimating processing in step S15168will be described later with reference to FIG. 143.

After the processing in step S15168, the flow proceeds to step S15169,and the image generating unit 15114 generates a signal more closelyapproximating the signal of the actual world 1, from the approximationfunction supplied from the actual world waveform estimating unit 15123,that is to say, generates an image with no movement blurring, and basedon the processing region information supplied from the processing regionsetting unit 15111, converts the portion of the processing region of theinput image into an image which has no movement blurring generated,generates this as an output image, and supplies this to the imagedisplay unit 15115, and the flow proceeds to step S15170.

In step S15170, the image display unit 15115 displays the output imagesupplied from the image generating unit 15114 instead of the input imagedisplayed in step S15161, or along with the input image, and the flowproceeds to step S15171.

In step S15171, as with the case of step S15162, the user I/F 15116determines whether or not there has been some sort of user input by theuser operating the user I/F 15116, and in the event that determinationis made that there has been no user input, i.e., in the event that theuser has made no operations, the flow returns to step S15171, and awaitssome sort of user input.

Also, in the event that determination is made in step S15171 that therehas been user input, i.e., in the event that the user has viewed theinput image or output image displayed on the image display unit 15115and operated the user I/F 15116, thereby making user input representingsome sort of instructions or information, the flow proceeds to stepS15172, where the user I/F 15116 determines whether or not the userinput is ending instructions instructing ending of the processing of thesignal processing device 4.

In the event that determination is made in step S15172 that the userinput is ending instructions, the signal processing device 4 ends theprocessing.

On the other hand, in the event that determination is made in stepS15172 that the user input is not ending instructions, the flow proceedsto step S15173, where the user I/F 15116 determines whether or not theuser input is constraint condition information. In the event thatdetermination is made in step S15173 that the user input is notconstraint condition information, the flow returns to step S15171, andhereafter the same processing is repeated.

Also, in step S15173, in the event that determination is made that theuser input is constraint condition information, the flow returns to stepS15165, and as described above, the user I/F 15116 supplies theconstraint condition information to the equation generating unit 15122.The flow then proceeds from step S15165 to step S15166, and hereafterthe same processing is repeated. In the event that the signal processingdevice 4 repeats the processing of step S15165 through step S15173, theprocessing of the above-described step S15166 and S15167 is the same asthat set by the first processing S15166 and S15167, and the same processas the first process may be repeated, or may be skipped.

Thus, with the signal processing device 4 in FIG. 140, by finding anapproximation function as a model wherein the signal of the actual world1 is modeled, an output image of high image quality with no movementblurring can be generated.

Also, by operating the user I/F 15116, the user can set constraintconditions which constrain the relation between each of the pixels inthe original image having no movement blurring, and in the event that anew constraint condition is input, an image without movement blurringcan be obtained again, and therefore an output image of high imagequality according to the preference of the user can be easily acquired.

With the signal processing device 4 in FIG. 140, an output image withoutmovement blurring is generated by finding an approximation functionwhich approximates the light signal of the actual world 1, but thisoutput image can be regarded as having the movement blurring removedfrom the input image. Accordingly, it can be said that processing isperformed for movement blurring removal with the signal processingdevice 4 in FIG. 140.

Next, the configuration of the inside of the actual world estimatingunit 15113 in FIG. 140 will be described with reference to FIG. 142.With the embodiment in FIG. 140 also, the mechanism for movementblurring generation of the object projected on the input image ismodeled as the relation of the above-described Expression (141),illustrated in FIG. 125 and FIG. 126 which is one example thereof.

In other words, let us say that the mechanism for moment blurringgeneration is modeled assuming that the pixel values of each of thepixels of the movement blurring image as the input image are valueswherein the pixel values (or the electric charge corresponding thereto)of each of the pixels of an image with no movement blurring which areaccumulated (integrated) while moving at the movement amount v.

The actual world estimating unit 15113 comprises a model generating unit15121, an equation generating unit 15122, and an actual world waveformestimating unit 15123. Further, the equation generating unit 15122comprises a constraint condition setting unit 15131, an activitydetecting unit 15132, a weighting changing unit 15133, and a normalequation generating unit 15134.

The movement amount v as the continuity information is supplied to themodel generating unit 15121 by the continuity setting unit 15112. Themovement amount v in this example expresses the continuity of the objectmoving in the horizontal direction at a constant speed of v pixels pershutter time, within the processing region of the input image which isset in the processing region setting unit 15111, and for example, let ussay that v=4, as with the above-described example. Also, the movementamount v may be a value which differs for each line, and in the eventthat the movement amount v differs for each line, k number, equal to thenumber of lines, of the movement amount v is supplied as continuityinformation from the continuity setting unit 15112 to the modelgenerating unit 15121.

Also, processing region information which specifies the processingregion within the input image is also supplied to the model generatingunit 15121 from the processing region setting unit 15111. In the case ofemploying a rectangular-shaped region as the processing region, theprocessing region information can be expressed, for example, bycoordinate values at opposite angles of a rectangular-shaped processingregion as to an absolute coordinate system wherein a predeterminedposition (pixel) of the input image is the origin, or the centercoordinate of the processing region and the width (horizontal direction)and the height (vertical direction) of the processing region.

The model generating unit 15121 acquires, for example, a processingregion width of l, and k line numbers, as illustrated in FIG. 119, fromthe processing region information supplied from the processing regionsetting unit 15111. In this example, let us say that the processingregion 15042 is rectangular-shaped, and for example, has l=10 and k=4.In this case, the processing region width of each line (the number ofpixels) are the same, but with the processing region 15042, each linecan have a different processing region width. In the case that theprocessing region width l differs for each line, the model generatingunit 15121 acquires a processing region width l of the same number (k)as the number of lines.

Also, the model generating unit 15121 generates an equation (modelequation) of a mixed pixel value P_(i) (i=3 through 9) which is shown inthe above-described Expression (141), based on the movement amount vsupplied from the continuity setting unit 15112 and the processingregion width l and k number of lines acquired from the processing regioninformation, and supplies this to the normal equation generating unit15134.

The constraint condition information specified by the user operating theuser I/F 15116 is supplied to the constraint condition setting unit15131 from the user I/F 15116. For example, the user performs anoperation with the user I/F 15116 to select (specify) the desiredconstraint condition from the constraint conditions which constrain therelation of the pixel values Q_(h) of each of the pixels of an imagewith no movement blurring, such as the above-described Expression (143)or Expression (144), which have been set in advance inside theconstraint condition setting unit 15131, and the user I/F 15116 suppliesthe information describing the selected constraint conditions as theconstraint condition information to the constraint condition settingunit 15131.

The constraint condition setting unit 15131 selects a constraintcondition expression corresponding to the constraint conditioninformation supplied from the user I/F 15116 from the constraintcondition expressions within the constraint condition setting unit 15131itself. Further, the constraint condition setting unit 15131 acquiresthe processing region width l based on the processing region informationof the input image supplied to the equation generating unit 15122 fromthe processing region setting unit 15111, and generates only the number,which corresponds to the processing region width l, of the constraintcondition expressions. For example, in the event that Expression (143)is employed which has the condition “adjacent pixel difference=0” as theconstraint condition information, the constraint condition setting unit15131 generates the constraint condition expression expressed by theabove-described Expression (145), by acquiring the processing regionwidth l based on the processing region information supplied from theprocessing region setting unit 15111, and supplies this to the normalequation generating unit 15134. In the example of the processing region15042 illustrated in FIG. 119, the processing region 15042 isrectangular-shaped, and the processing region width l is the same ofeach line, but in the case that the processing region width l differsfor each line, the number of constraint condition expressions alsodiffer for each line.

With the activity detecting unit 15132, the activity (features) of eachof the pixels (pixel of interest) within the processing region of theinput image, which is to become information for the purpose of theweighting changing unit 15133 to determine the weighting W_(mi) as tothe equation of each of the mixed pixels P_(i) (model equation) and theweighting W_(bj) as to each of the constraint condition expressions, isdetected.

In other words, the activity detecting unit 15132 detects the value ofthe adjacent pixel difference (P_(j)−P_(j+1)) between the value of thepixel of interest P_(j) and the pixel P_(j+1) adjacent to the rightthereof, or the like, for example, with each of the pixels within theprocessing region of the input image as a pixel of interest P_(j), forexample, as activity of the pixel of interest P_(j), and supplies thisto the weighting changing unit 15133.

The weighting changing unit 15133 has a weighting W_(mi) (hereafter willbe simply called weighting W_(mi)) as to the equation for each of themixed pixel values P_(i) (the model equation) and a weighting W_(bj)(hereafter will be simply called weighting W_(bj)) as to each of theconstraint condition expressions.

By supplying the weighting W_(mi) and the weighting W_(bj) to the normalequation generating unit 15134, the weighting changing unit 15133changes (sets) the weighting of each of the equations of the mixed pixelvalue P_(i) equation supplied from the model generating unit 15121 (themodel equation), and the equation formed from the constraint conditionexpressions supplied from the constraint condition setting unit 15131,which are generated in the normal equation generating unit 15134.

The deterioration of the image comprised by the pixel value Q_(h)acquired with the actual world estimating unit 15113 can be conceived tooccur often in positions where the constraint condition isinappropriate, and thus, adjusting the weighting W_(bj) appropriately ismore effective than the weighting W_(mi). Thus, with this embodiment,the weighting W_(mi) is a predetermined value (fixed value), and theweighting W_(bj) is changed based on the activity supplied from theactivity detecting unit 15132. Now, the weighting W_(mi) may also bemade to be changed by the activity found with similar methods to bedescribed below, or by other methods.

In the event that the “adjacent pixel difference=0”expressed by theabove-described Expression (143), for example, is employed as aconstraint condition expression, the positions (pixels) wherein theconstraint condition expression expressed by Expression (143) is notestablished are pixels corresponding to the edge portions of the objectin the image with no movement blurring. Here, the constraint conditionexpression expressed by Expression (143) considers the differencebetween adjacent pixels in the direction of the spatial direction X, andtherefore, the edge portion of the object wherein the constraintcondition expression is not established is the edge in the directionthat the object edge direction (border line) is not in the horizontaldirection (a parallel direction to the spatial direction X).

The edge portion of the image with no movement blurring occurring isalso the edge in an image with movement blurring. In other words, apixel which was an edge portion in an image with no movement blurringoccurring still has the characteristics of the edge in the vicinity ofthis pixel even in an image with movement blurring occurring.Accordingly, the weighting W_(bj) can be determined according to thevalue of the adjacent pixel difference (P_(j)−P_(j+1)) of the image(input image) wherein movement blurring is occurring.

In other words, positions with a large value of the adjacent pixeldifference (P_(j)−P_(j+1)) of the input image has a greater probabilityof being an edge in an image with no movement blurring occurring. Thus,the weighting W_(bj) of the constraint condition expression in theabove-described Expression (143) wherein the adjacent pixel differenceis small is made to be smaller (the constraint condition is relaxed).There are two methods as will be described below, for example, asmethods for causing the weighting W_(bj) to be smaller corresponding tothe adjacent pixel difference (P_(j)−P_(j+1)) which shows activity.

The first method is a method employing the inverse ratio of the value ofthe adjacent pixel difference (P_(j)−P_(j+1)) of the input image as theweighting W_(bj). This can be expressed as the following Expression.

$\begin{matrix}{{W_{bj} = {{\frac{1}{P_{j} - P_{j + 1}}\mspace{14mu} j} = 0}},\ldots\mspace{11mu},8} & (159)\end{matrix}$

The second method is a method setting the weighting W_(bj) of theconstraint condition as either 0 or 1, according to the value of theadjacent pixel difference (P_(j)−P_(j+1)) of the input image, andequivalently determining whether or not to include the constraintcondition expression. This can be expressed as the following Expression.

$\begin{matrix}\begin{matrix}{W_{bj} = \left\{ \begin{matrix}0 & \left( {{P_{j} - P_{j + 1}} > {Th}} \right) \\1 & \left( {{P_{j} - P_{j + 1}} < {Th}} \right)\end{matrix} \right.} \\{{j = 0},\cdots\mspace{11mu},8}\end{matrix} & (160)\end{matrix}$

Here, in Expression (160), the threshold value Th is a value which isthe basis for determining whether or not to include the constraintcondition expression, and can be set in advance in the weightingchanging unit 15133. The constraint condition expression wherein theweighting W_(bj) is set as 0 is essentially removed from the conditionconstraint expression.

The equation of the mixed pixel value P_(i) having the same number k asthe number of lines (Expression (141)) is supplied to the normalequation generating unit 15134 from the model generating unit 15121.Also, the constraint condition Expression (Expression (145))corresponding to the constraint condition selected by the user operatingthe user I/F 15116 is supplied to the normal equation generating unit15134 from the constraint condition setting unit 15132.

With the normal equation generating unit 15134, the weighting W_(bj) isset for Expression (147) wherein the error e_(bj) is considered to theconstraint condition Expression (145) supplied from the constraintcondition setting unit 15132, by the weighting changing unit 15133 (theweighting W_(bj) as to the model equation and the constraint conditionwhich was has been 1:1 is changed).

In other words, Expression (147) is changed to the following Expression(161) which has multiplied both sides thereof by W_(bj).

$\begin{matrix}\begin{matrix}{{W_{b\; 0}\left( {Q_{0} - Q_{1}} \right)} = {W_{b\; 0}e_{b\; 0}}} \\{{W_{b\; 1}\left( {Q_{1} - Q_{2}} \right)} = {W_{b\; 1}e_{b\; 1}}} \\{{W_{b\; 2}\left( {Q_{2} - Q_{3}} \right)} = {W_{b\; 2}e_{b\; 2}}} \\{{W_{b\; 3}\left( {Q_{3} - Q_{4}} \right)} = {W_{b\; 3}e_{b\; 3}}} \\{{W_{b\; 4}\left( {Q_{4} - Q_{5}} \right)} = {W_{b\; 4}e_{b\; 4}}} \\{{W_{b\; 5}\left( {Q_{5} - Q_{6}} \right)} = {W_{b\; 5}e_{b\; 5}}} \\{{W_{b6}\left( {Q_{6} - Q_{7}} \right)} = {W_{b\; 6}e_{b\; 6}}} \\{{W_{b\; 7}\left( {Q_{7} - Q_{8}} \right)} = {W_{b\; 7}e_{b\; 7}}} \\{{W_{b\; 8}\left( {Q_{8} - Q_{9}} \right)} = {W_{b\; 8}e_{b\; 8}}}\end{matrix} & (161)\end{matrix}$

Also, with the normal equation generating unit 15134, the weightingW_(mi) is set for Expression (146) wherein the error e_(mi) isconsidered to the mixed pixel value P_(i) equation (model equation)(141) supplied from the model generating unit 15121, by the weightingchanging unit 15133.

In other words, Expression (146) is changed to the following Expression(162) which has multiplied both sides thereof by the weighting W_(mi).

$\begin{matrix}{{{W_{m\; 3} \times {\left( {Q_{0} + Q_{1} + Q_{2} + Q_{3}} \right)/v}} = {W_{m\; 3} \times \left( {P_{3} + e_{m\; 3}} \right)}}{{W_{m\; 4} \times {\left( {Q_{1} + Q_{2} + Q_{3} + Q_{4}} \right)/v}} = {W_{m\; 4} \times \left( {P_{4} + e_{m\; 4}} \right)}}{{W_{m\; 5} \times {\left( {Q_{2} + Q_{3} + Q_{4} + Q_{5}} \right)/v}} = {W_{m\; 5} \times \left( {P_{5} + e_{m\; 5}} \right)}}{{W_{m\; 6} \times {\left( {Q_{3} + Q_{4} + Q_{5} + Q_{6}} \right)/v}} = {W_{m\; 6} \times \left( {P_{6} + e_{m\; 6}} \right)}}{{W_{m\; 7} \times {\left( {Q_{4} + Q_{5} + Q_{6} + Q_{7}} \right)/v}} = {W_{m\; 7} \times \left( {P_{7} + e_{{m\; 7}\;}} \right)}}{{W_{m\; 8} \times {\left( {Q_{5} + Q_{6} + Q_{7} + Q_{8}} \right)/v}} = {W_{m8} \times \left( {P_{8} + e_{m\; 8}} \right)}}{{W_{m\; 9} \times {\left( {Q_{6} + Q_{7} + Q_{8} + Q_{9}} \right)/v}} = {W_{m\; 9} \times \left( {P_{9} + e_{m\; 9}} \right)}}} & (162)\end{matrix}$

As described above, the weighting W_(mi) in Expression (162) is a fixedvalue, here.

As illustrated in FIG. 128, the normal equation generating unit 15134selects one predetermined line from the processing region 15042 as theline of interest. Then the normal equation generating unit 15134generates a normal equation expressed with the following Expression(163) from Expression (161) and Expression (162) corresponding to theselected line of interest.

$\begin{matrix}{{{\left\lbrack \frac{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}W_{m\; 3} \\W_{m\; 4}\end{matrix} \\W_{m\; 5}\end{matrix} \\W_{m\; 6}\end{matrix} \\W_{m\; 7}\end{matrix} \\W_{m\; 8}\end{matrix} \\W_{m\; 9}\end{matrix}}{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}W_{b\; 0} \\W_{b\; 1}\end{matrix} \\W_{b\; 2}\end{matrix} \\W_{b\; 3}\end{matrix} \\W_{b\; 4}\end{matrix} \\W_{b\; 5}\end{matrix} \\W_{b\; 6}\end{matrix} \\W_{b\; 7}\end{matrix} \\W_{b\; 8}\end{matrix}} \right\rbrack^{T}\left\lbrack \frac{\begin{matrix}{1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & {1/v} & {1/v} & {1/v} & {1/v}\end{matrix}}{\begin{matrix}1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & {- 1}\end{matrix}} \right\rbrack}\begin{bmatrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}Q_{0} \\Q_{1}\end{matrix} \\Q_{2}\end{matrix} \\Q_{3}\end{matrix} \\Q_{4}\end{matrix} \\Q_{5}\end{matrix} \\Q_{6}\end{matrix} \\Q_{7}\end{matrix} \\Q_{8}\end{matrix} \\Q_{9}\end{bmatrix}} = {\left\lbrack \frac{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}W_{m\; 3} \\W_{m\; 4}\end{matrix} \\W_{m\; 5}\end{matrix} \\W_{m\; 6}\end{matrix} \\W_{m\; 7}\end{matrix} \\W_{m\; 8}\end{matrix} \\W_{m\; 9}\end{matrix}}{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}W_{b\; 0} \\W_{b\; 1}\end{matrix} \\W_{b\; 2}\end{matrix} \\W_{b\; 3}\end{matrix} \\W_{b\; 4}\end{matrix} \\W_{b\; 5}\end{matrix} \\W_{b\; 6}\end{matrix} \\W_{b\; 7}\end{matrix} \\W_{b\; 8}\end{matrix}} \right\rbrack^{T}\left( {\left\lbrack \frac{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}P_{3} \\P_{4}\end{matrix} \\P_{5}\end{matrix} \\P_{6}\end{matrix} \\P_{7}\end{matrix} \\P_{8}\end{matrix} \\P_{9}\end{matrix}}{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}0 \\0\end{matrix} \\0\end{matrix} \\0\end{matrix} \\0\end{matrix} \\0\end{matrix} \\0\end{matrix} \\0\end{matrix} \\0\end{matrix}} \right\rbrack + \left\lbrack \frac{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}e_{m\; 3} \\e_{m\; 4}\end{matrix} \\e_{m\; 5}\end{matrix} \\e_{m\; 6}\end{matrix} \\e_{m\; 7}\end{matrix} \\e_{m\; 8}\end{matrix} \\e_{m\; 9}\end{matrix}}{\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}e_{b\; 0} \\e_{b\; 1}\end{matrix} \\e_{b\; 2}\end{matrix} \\e_{b\; 3}\end{matrix} \\e_{b\; 4}\end{matrix} \\e_{b\; 5}\end{matrix} \\e_{b\; 6}\end{matrix} \\e_{b\; 7} \\e_{b\; 8}\end{matrix}} \right\rbrack} \right)}} & (163)\end{matrix}$

Here, Expression (163) is replaced with Expression (164), using thematrix A″ and the column vectors x, y, e″.

$\begin{matrix}{{A^{''}x} = {y + e^{''}}} & (164)\end{matrix}$

In this case, the total sum E of the squared errors of the error e_(bj)of Expression (161) and the error e_(mi) of Expression (162) can beexpressed with the following expression.

$\begin{matrix}{E^{''} = {{\sum\limits_{\;}^{\;}\left( {W_{m\; i}e_{m\; i}} \right)^{2}} + {\sum\left( {W_{bj}e_{bj}} \right)^{2}}}} & (165)\end{matrix}$

In order to find the Q_(h) which causes the total sum E″ of the squarederror to be very small, the column vector x (the column vector x withthe pixel value Q_(h) as a component thereof) can be expressed withExpression (166) from the same expression as the above Expression (150)and Expression (151).

$\begin{matrix}{x = {\left( {A^{''T}A^{''}} \right)^{- 1}A^{''T}y}} & (166)\end{matrix}$

Now, with Expression (166), the superscript T means transposing, and thesuperscript −1 means an inverse matrix.

Accordingly, the normal equation generating unit 15134 computes thematrix (A″^(T)A″)⁻¹A″^(T), which is a coefficient of the column vector yof the right side of Expression (166), and substitutes the mixed pixelvalue P_(i) which is acquired based on the input image supplied from thesensor 2 (FIG. 1) into the column vector y. The matrix(A″^(T)A″)⁻¹A″^(T) is computed, and Expression (166) into which themixed pixel value P_(i) is substituted is supplied from the normalequation generating unit 15134 to the actual world waveform estimatingunit 15123.

When the processing such as described above is finished wherein onepredetermined line of the processing region 15042 is selected as theline of interest, the normal equation generating unit 15134 selects aline not yet selected as the next line of interest, and similarprocessing is performed as to the selected next line of interest.

The actual world waveform estimating unit 15123, by computing Expression(166) which is supplied from the normal equation generating unit 15134,finds the column vector x, that is to say, the pixel value Q_(h) with nomovement blurring, and supplies this to the image generating unit 15114(FIG. 140).

The image generating unit 15114 replaces a portion of the processingregion 15042 of the input image with the image of the pixel value Q_(h)with no movement blurring, which is supplied from the actual worldwaveform estimating unit 15123, based on the processing regioninformation supplied from the processing region setting unit 15111, andgenerates this as an output image, and supplies this to the imagedisplay unit 15115.

Next, the actual world estimating processing of the actual worldestimating unit 15113 in step S15168 in FIG. 141 will be described withreference to the flowchart in FIG. 143.

First, in step S15191, the model generating unit 15121 acquires theprocessing region width l and the number of lines k, based on theprocessing region information supplied from the processing regionsetting unit 15111, and the flow proceeds to step S15192.

In step S15192, the model generating unit 15121 acquires a k number ofmovement amount v, which is equivalent to the number of lines in theprocessing region 15042, from the continuity setting unit 15112, and theflow proceeds to step S15193.

In step S15193, the model generating unit 15121 generates an equationfor the mixed pixel value P_(i) for each line, and the flow proceeds tostep S15194. In other words, the model generating unit 15121 generates knumber of model equations (expressions (141)).

In step S15194, the constraint condition setting unit 15131 selects theconstraint condition expression corresponding to the constraintcondition information supplied from the user I/F 15116, from theconstraint condition expressions existing within the constraintcondition setting unit itself. Also, the constraint condition settingunit 15131 acquires the processing region width l from the processingregion information of the input image which is supplied from theequation generating unit 15122 from the processing region setting unit15111, and generates an constraint condition Expression (Expression(145)) corresponding to the constraint condition information from theusers interface 15116, which comprises a number of equationscorresponding to the processing region width l, and the flow proceeds tostep S15195.

In step S15195, the activity detecting unit 15132 detects the activity(features) of each of the pixels within the processing region of theinput image which is information for the weighting changing unit 15133to determine the weighting W_(bj), and supplies this to the weightingchanging unit 15133, and the flow proceeds to step S15196.

In step S15196, the weighting changing unit 15133 calculates theweighting W_(bj) to be found with, for example, the above describedfirst or second method, using the activity of each of the pixels withinthe processing region of the input image, which is supplied from theactivity detecting unit 15132. Also, the weighting changing unit 15133sets the calculated weighting W_(bj) and the weighting W_(mi) providedin advance as a fixed value as the weighting respectively for each ofthe equations of the mixed pixel value P_(i) equation of the normalequation generating unit 15134 (model equation) and the expression forconstraint conditions (constraint condition expression), and the flowproceeds to step S15197.

In step S15197, the normal equation generating unit 15134 selects onepredetermined line from the processing region 15042 as a line ofinterest, and the flow proceeds to step S15198.

In step S15198, the normal equation generating unit 15134 generates anormal equation expressed by Expression (163) (Expression (164)), fromExpression (162) wherein the weighting W_(mi) is changed by theweighting changing unit 15133 in step S15196, as to Expression (146)which considers the error e_(mi) of each of the expressions in theequation of the mixed pixel value P_(i) expressed with Expression (141),which is supplied from the model generating unit 15121, and theconstraint condition Expression (161) wherein the weighting W_(bj) ischanged by the weighting changing unit 15133 in step S15196, as toExpression (147) which considers the error e_(bj) of each of theexpressions in the constraint condition Expression (145), which issupplied from the constraint condition setting unit 15131, and the flowproceeds to step S15199.

In step S15199, the normal equation generating unit 15134 computes thematrix (A″^(T)A″)⁻¹A″^(T), which is a coefficient of the column vector yof the right side of Expression (166), and the flow proceeds to stepS15200.

In step S15200, the normal equation generating unit 15134 substitutesthe mixed pixel value P_(i) which is acquired based on the input imagesupplied from the sensor 2 (FIG. 1) into the column vector y, and theflow proceeds to step S15201.

In step S15201, the actual world waveform estimating unit 15123 computesa normal equation. That is to say, the actual world waveform estimatingunit 15123 finds a pixel value Q_(h) with no movement blurring bycomputing Expression (166) which is supplied from the normal equationgenerating unit 15134, and the flow proceeds to step S15202.

In step S15202, determination is made as to whether or not processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042. In the event that it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15197, and the processing insteps S15197 through S15202 is repeated. That is to say, a line not yetselected of the processing region 15042 is selected as the next line ofinterest, and processing is performed to find a pixel value Q_(h) withno movement blurring.

On the other hand, in the event that it is determined that processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042, the actual world estimatingprocessing ends and returns.

Thus, by generating a normal equation (Expression (163)) as to one linewhich is the line of interest of the processing region 15042, andrepeating the processing to find a pixel value Q_(h) with no movementblurring k number of times, which is the number of lines in theprocessing region 15042, the pixel value Q_(h) with no movement blurringcan be found across the entire region of the processing region 15042.

Thus, with the embodiment illustrated in FIG. 140, a normal equation isgenerated from an equation of the mixed pixel value P_(i) wherein apixel value generating movement blurring is modeled with a pixel valuewith no movement blurring, and a constraint condition expression usingthe relation between adjacent pixels of a pixel value with no movementblurring, and by computing this normal equation, a pixel value with nomovement blurring is found, and therefore an image with good imagequality can be generated.

Also, the weighting of each of the constraint conditions expressions ischanged based on the activity of the input image, and therefore an imagewith good image quality can be generated according to the features ofthe input image.

FIGS. 144 through 146 are diagrams describing the processing results ofthe signal processing device 4 in FIG. 140.

FIG. 144 illustrates an input image to be input in the signal processingdevice 4 in FIG. 140. With the input image in FIG. 144, the characters“TAL” are moving in the direction from left to right at the same speed,and movement blurring is occurring in the horizontal direction.

FIG. 145 illustrates an output image wherein the input image in FIG. 144has been processed by the signal processing device 4 in FIG. 135. Thatis to say, this is an output image as to the input shown in FIG. 144,wherein the weighting W_(bj) regarding the constraint conditions isapplied to all constraint condition expressions as the same weight. Asis clear by comparing FIG. 144 and FIG. 145, in the image in FIG. 145,the movement blurring occurring in the input image in FIG. 144 isreduced, and the characters “TAL” are relatively clearly displayed.

On the other hand, FIG. 146 illustrates the output image wherein theinput image in FIG. 144 has been processed by the signal processingdevice 4 in FIG. 140 which is the above-described embodiment, using theabove-described second method. That is, this is an output image as tothe input shown in FIG. 144, wherein the weighting W_(bj) regarding theconstraint condition expressions is set to 0 or to 1 depending on theactivity. In the image in FIG. 146 also, the movement blurring occurringin the input image in FIG. 144 is reduced, and the characters “TAL” areclearly displayed.

Now, with the embodiment in FIG. 144 through FIG. 146, the processedimage in FIG. 146 is an image wherein the edge portion is clearer thanthe processed image in FIG. 145.

Also, in the above-described example, the constraint conditioninformation supplied from the user I/F 15116 to the constraint conditionsetting unit 15131 is said to be information showing which constraintcondition the user has selected (specified) from the multiple constraintconditions set in advance inside the constraint condition setting unit15131, but the user may directly input the constraint conditionexpression by operating the user I/F 15116.

Also, in the case that the above-described second method is used withthe weighting changing unit 15133, the threshold value Th which is avalue that is a basis for determining whether or not to input aconstraint condition expression is said to be set in advance internally,but the user can also adjust the threshold value Th as appropriate byoperating the user I/F 15116 while viewing the processing imagedisplayed on the image display unit 15115. FIG. 147 illustrates aconfiguration example of another embodiment of an application example ofthe signal processing device 4 in FIG. 111.

An image of, for example, one frame or one field as the data 3 from thesensor 2 is input into the signal processing device 4 in FIG. 147. Here,let us say that an object having a predetermined shape is displayed inthe input image, such object moving in the horizontal direction(crosswise direction) of the image at a constant speed of v pixels pershutter time (exposure time). In other words, regarding the input image,the object is moving in the horizontal direction at the movement amountof v pixels, and therefore because of the time integration effect of thesensor 2, the light signal of the object and the light signal of theportion other than the object are mixed (time mixture), and thus, theimage is blurred at portions such as the borders of the object. Thus,with the signal processing device 4 in FIG. 147, an output image of highimage quality with no movement blurring can be generated by such timemixture.

In FIG. 147, the processing region setting unit 15151, the continuitysetting unit 15152, the actual world estimating unit 15153, imagegenerating unit 15154, image display unit 15155, and user I/F 15156,each correspond to the processing region setting unit 10001, thecontinuity setting unit 10002, the actual world estimating unit 10003,image generating unit 10004, image display unit 10005, and user I/F10006 shown in FIG. 111, and basically perform the same processing asthe processing region setting unit 10001, the continuity setting unit10002, the actual world estimating unit 10003, image generating unit10004, image display unit 10005, and user I/F 10006. Further, in FIG.147, the actual world estimating unit 15153 comprises a model generatingunit 15161, an equation generating unit 15162, and actual world waveformestimating unit 15163. The model generating unit 15161, equationgenerating unit 15162, and actual world waveform estimating unit 15163correspond to each of the model generating unit 10011, equationgenerating unit 10012, and actual world waveform estimating unit 10013in FIG. 111, and basically perform the same functions as each of themodel generating unit 10011, equation generating unit 10012, and actualworld waveform estimating unit 10013.

Note however, that in FIG. 147, the assisting information which the userI/F 15156 outputs by the user operating the user I/F 15156, is suppliedonly to the equation generating unit 15162.

In other words, in FIG. 147, by operating the user I/F 15116, with theequation generating unit 15162, the user can set the constraintconditions which constrain the relation between each of the pixels of animage with no movement blurring, which is an image of the actual world 1corresponding to the input image and the weighting as to the constraintcondition thereof, and when the user performs an operation to set theconstraint conditions or an operation to set the weighting, the user I/F15156 supplies the constraint condition information expressing theconstraint conditions set by the operation or the weighting as to theconstraint condition to the equation generating unit 15162.

Specifically, for example, a user who views an image displayed on theimage display unit 15155 reflecting a blurring image guesses theoriginal image, that is to say, an image without movement blurring whichis an image of the actual world 1 corresponding to the input image.Also, for example, the user guesses predetermined regions where blurringis occurring as the edge portions, and guesses the relation of the pixelvalues between each of the pixels (the constraint conditions), such asthe difference being large between the pixel values (level) between theadjacent pixels at the edge portions. Further, the user can set theconditions suitable for the guessed relation of the pixels value betweeneach of the pixels into the equation generating unit 15162, by operatingthe user I/F 15156.

Also, in FIG. 147, the processing region information, which is suppliedfrom the processing region setting unit 15151 to the actual worldestimating unit 15153, is supplied to the model generating unit 15161and the equation generating unit 15162 of the actual world estimatingunit 15153. The model generating unit 15161 and the equation generatingunit 15162 recognize the processing region in the input image from theprocessing region information supplied from the processing regionsetting unit 15151.

The continuity setting unit 15152 sets a movement amount (movementvector) v in the input image as the continuity information that anobject is moving in the horizontal direction of the image at theconstant speed of v pixels per shutter time, and supplies this movementamount v as the continuity information to the model generating unit15161 of the actual world estimating unit 15153.

The model generating unit 15161 generates an equation (hereafterreferred to as model equation as appropriate) as a model (hereafterreferred to as relation model as appropriate), modeling the relationbetween the signal of the actual world 1 and the pixel values of each ofthe pixels of the movement blurring image as the input image whichconsiders the movement amount v as the continuity information suppliedfrom the continuity setting unit 15152, regarding the processing regionrecognized by the processing region information which is supplied fromthe processing region setting unit 15151, and supplies the generatedmodel equation to the equation generating unit 15162.

The equation generating unit 15162 generates a constraint conditionExpression (hereafter referred to as constraint condition expression asappropriate), based on the constraint condition information suppliedfrom the user I/F 15156. Also, the equation generating unit 15162 setsthe weighting supplied from the user I/F 15156 into the constraintcondition expression and the model equation supplied to the equationgenerating unit 15162 by the model generating unit 15161, andsubstitutes the pixel values of each of the pixels of the input imagecomprising the processing region into the equation formed from theweighted constraint condition expression and the model equation, andthus, generates an equation to find the approximation function as amodel (hereafter referred to as approximation model) wherein the signalof the actual world 1 is modeled, and supplies this to the actual worldwaveform estimating unit 15163.

The actual world waveform estimating unit 15163 estimates the waveformof the actual world 1 signal, by computing the equation supplied fromthe equation generating unit 15162. In other words, the actual worldwaveform estimating unit 15163 finds the approximation function as anapproximation model, and supplies the approximation function to theimage generating unit 15154 as the estimation result of the signalwaveform of the actual world 1. Note that approximation functionapproximating the actual world 1 signals include functions with constantfunction values, regardless of argument value.

The image generating unit 15154 recognizes the processing region in theinput image, based on the processing region information supplied fromthe processing region setting unit 15151. Also, the image generatingunit 15154 generates a signal more closely approximating the signal ofthe actual world 1, from the approximation function supplied from theactual world waveform estimating unit 15163, that is to say, generatesan image with no movement blurring, and replaces the image of theprocessing region of the input image with an output image wherein theimage has no movement blurring, generates this as an output image, andsupplies this to the image display unit 15155.

Next, the processing of the signal processing device 4 in FIG. 147 willbe described with reference to the flowchart in FIG. 148.

First, in step S15211, the signal processing device 4 performspre-processing, and the flow proceeds to step S15212. That is to say,the signal processing device 4 supplies one frame or one field, forexample, of the input image, supplied from the sensor 2 (FIG. 1) as data3, to the processing region setting unit 15151, continuity setting unit15152, actual world estimating unit 15153, image generating unit 15154,and image display unit 15155. Further, the signal processing unit 4causes the image display unit 15155 to display the input image.

In step S15212, the user I/F 15156 determines whether or not there hasbeen some sort of user input, by the user operating the user I/F 15156.In step S15212, in the event that determination is made that there is nouser input, i.e., in the event that the user has made no operations, theflow skips steps S15213 through S15215, and proceeds to step S15216.

On the other hand, in step S15212, in the event that determination ismade that there has been user input, i.e., in the event that the userhas viewed the input image displayed on the image display unit 15155 andoperated the user I/F 15156, thereby making user input indicating somesort of instruction or information, and in the event that there has beensome sort of user input, the flow proceeds to step S15213, where theuser I/F 15156 determines whether or not the user input is userinstructions instructing ending of the processing of the signalprocessing device 4.

In the event that determination is made that in step S15213 that theuser input is ending instructions, the signal processing device 4 endsthe processing.

On the other hand, in the event that determination is made in stepS15213 that the user input is not ending instructions, the flow proceedsto step S15214, where the user I/F 15156 determines whether or not theuser input is constraint condition information or weighting. In theevent that determination is made in step S15214 that the user input isnot constraint condition information nor weighting, step S15215 isskipped, and the flow proceeds to step S15216.

Also, in step S15214, in the event that the user input is determined tobe constraint condition information or weighting, the flow proceeds tostep S15215, and the user I/F 15156 supplies the constraint conditioninformation or weighting to the equation generating unit 15162, and theflow proceeds to step S15216. With the user I/F 15156, both theconstraint condition information and the weighting can be input, and inthe event that the user inputs both the constraint condition informationand the weighting, the user I/F 15156 supplies both the constraintcondition information and the weighting to the equation generating unit15162.

In step S15216, the processing region setting unit 15151 sets theprocessing region based on the input image, and supplies the processingregion information identifying the processing region to the continuitysetting unit 15152, the model generating unit 15161 and equationgenerating unit 15162 of the actual world estimating unit 15153, andimage generating unit 15154, and the flow proceeds to step S15217. Now,an arrangement maybe made wherein the setting of the processing regionin step S15216 is performed by the user inputting processing regioninstruction information by operating the user I/F 15156, so as to becarried out based on the processing region instruction information, or,this may be carried out without processing region instructioninformation from the user.

In step S15217, the continuity setting unit 15152 recognizes theprocessing region in the input image, from the processing regioninformation supplied from the processing region setting unit 15151.Further, the continuity setting unit 15152 sets the continuity of theactual world 1 signals of which a portion of the continuity is lost inthe processing region image data, and supplies continuity informationrepresenting the continuity for the processing region to the modelgenerating unit 15161 of the actual world estimating unit 15153, and theflow proceeds to step S15218. Here, the continuity setting unit 15152sets the movement amount v, expressing that the movement is in thehorizontal direction at a constant speed of v pixels per shutter time(exposure time), as the continuity information, and supplies this to themodel generating unit 15161 of the actual world estimating unit 15153.Here, the continuity setting unit 15152 sets the movement amount showingonly the movement size as continuity information, under the premise thatthe object is moving in a horizontal direction, but a movement vectorshowing the size and direction of the movement of the object can be setas continuity information as well. The setting of continuity in stepS15217 can be arranged by the user inputting the continuity instructioninformation by operating the user I/F 15156, and can be performed basedon this continuity instruction information, or can be performed with nocontinuity instruction information from the user.

The actual world estimating unit 15153 performs actual world estimatingprocessing in step S15218. In other words, with the actual worldestimating unit 15153, the model generating unit 15161 generates anequation (model equation) as the model (relation model) wherein therelation of the pixel values of each of the pixels of the input imagewhich has movement blurring and the signal of the actual world 1 ismodeled, based on the movement amount v supplied from the continuitysetting unit 15152 in step S15217 and the processing region informationsupplied from the processing region setting unit 15151 in step S15216,and the generated model equation is supplied to the equation generatingunit 15162.

The equation generating unit 15162 generates a constraint conditionexpression based on the constraint condition information supplied fromthe user I/F 15156 in step S15215, and sets the weighting supplied fromthe user I/F 15156 into the generated constraint condition expressionand into the model equation supplied from the model generating unit15161 to the equation generating unit 15162. Also, the equationgenerating unit 15162 substitutes the pixel values of each of the pixelsof the input image into an equation formed from the weighted constraintcondition expression and the model equation, and thus generates anequation for finding the approximation function as the model(approximation model) wherein the signal of the actual world 1 ismodeled, and supplies this to the actual world waveform estimating unit15163. In the event that the user has not specified (set) constraintcondition information, that is to say, in the event that NO isdetermined to be the case in steps S15212 or S15214, the equationgenerating unit 15162 performs predetermined processing such as a methodfor solving a simultaneous equation formed of a model equation and anequation hypothesizing that the edge portion of the processing region is“flat”, as described in the embodiment in FIG. 113.

Now, the weighting can be set for the model equation only, for theconstraint condition expression only, or for both the constraintcondition expression and the model equation.

The actual world waveform estimating unit 15163 estimates the waveformof the actual world 1 signal, that is to say, obtains an approximationfunction approximating the actual world 1 signals by solving theequation supplied from the equation generating unit 15162, and suppliesthe approximation function to the image generating unit 15154, asestimation results of the waveform of the actual world 1 signals.

The details of the actual world estimating process of step S15218 willbe described later with reference to FIG. 150.

After the processing in step S15218, the flow proceeds to step S15219,and the image generating unit 15154 generates a signal more closelyapproximating the signal of the actual world 1, from the approximationfunction supplied from the actual world waveform estimating unit 15163,that is to say, generates an image with no movement blurring, and basedon the processing region information supplied from the processing regionsetting unit 15151, replaces the portion of the processing region of theinput image with an image which has no movement blurring generated,generates this as an output image, and supplies this to the imagedisplay unit 15155, and the flow proceeds to step S15220.

In step S15220, the image display unit 15155 displays the output imagesupplied from the image generating unit 15154 instead of the input imagedisplayed in step S15211, or along with the input image, and the flowproceeds to step S15221.

In step S15221, as with the case of step S15212, the user I/F 15156determines whether or not there has been some sort of user input by theuser operating the user I/F 15156, and in the event that determinationis made that there has been no user input, i.e., in the event that theuser has made no operations, the flow returns to step S15221, and awaitssome sort of user input.

Also, in the event that determination is made in step S15221 that therehas been user input, i.e., in the event that the user has viewed theinput image or output image displayed on the image display unit 15155and operated the user I/F 15156, thereby making user input representingsome sort of instructions or information, the flow proceeds to stepS15222, where the user I/F 15156 determines whether or not the userinput is ending instructions instructing ending of the processing of thesignal processing device 4.

In the event that determination is made in step S15222 that the userinput is ending instructions, the signal processing device 4 ends theprocessing.

On the other hand, in the event that determination is made in stepS15222 that the user input is not ending instructions, the flow proceedsto step S15223, where the user I/F 15156 determines whether or not theuser input is constraint condition information or weighting. In theevent that determination is made in step S15223 that the user input isnot constraint condition information nor weighting, the flow returns tostep S15221, and hereafter the same processing is repeated.

Also, in the event that determination is made in step S15223 that theuser input is constraint condition information or weighting, the flowreturns to step S15215, and as described above, the user I/F 15156supplies the constraint condition information or weighting to theequation generating unit 15162. Then the flow proceeds from step S15215to S15216, and hereafter the same processing is repeated. Now, in theevent that the signal processing device 4 repeats the processing of stepS15215 through step S15223, the processing of the above-described stepS15216 and S15217 are the same as that set by the first processingS15216 and S15217, and the same process as the first process may berepeated, or may be skipped.

Thus, with the signal processing device 4 in FIG. 147, by finding anapproximation function as a model wherein the signal of the actual world1 is modeled, an output image of high image quality with no movementblurring can be generated.

Also, by operating the user I/F 15156, the user can set constraintconditions which constrain the relation between each of the pixels inthe original image having no movement blurring, and in the event that anew constraint condition is input, an image without movement blurringcan be requested again, and therefore an output image of high imagequality according to the preference of the user can be easily acquired.

In the signal processing device 4 in FIG. 147, an output image withoutmovement blurring is generated by finding an approximation functionwhich approximates the light signal of the actual world 1, but thisoutput image can be regarded as having the movement blurring removedfrom the input image. Accordingly, it can be said that processing isperformed for movement blurring removal with the signal processingdevice 4 in FIG. 147.

Next, the internal configuration of the actual world estimating unit15153 in FIG. 147 will be described with reference to FIG. 149. With theembodiment in FIG. 147 also, the mechanism for generating movementblurring of objects reflecting in the input image is modeled as arelation of the above-described Expression (141), which is an exampleshown in FIGS. 125 and 126.

In other words, let us say that the mechanism for movement blurringgenerating is modeled assuming that the pixel values of each of thepixels of the movement blurring image as the input image are valueswherein the pixel values (or the electric charge corresponding thereto)of each of the pixels of an image with no movement blurring which areaccumulates (integrated) while moving at the movement amount v.

The actual world estimating unit 15153 comprises a model generating unit15161, an equation generating unit 15162, and an actual world waveformestimating unit 15163. Further, the equation generating unit 15162comprises a constraint condition setting unit 15171, a weightingchanging unit 15172, and a normal equation generating unit 15173.

The movement amount v as the continuity information is supplied to themodel generating unit 15161 by the continuity setting unit 15152. Themovement amount v in this example expresses the continuity of the objectmoving in the horizontal direction at a constant speed of v pixels pershutter time, within the processing region of the input image which isset by the processing region setting unit 15151, and for example, let ussay that v=4, as with the above-described example. Also, the movementamount v can be a value which differs for each line, and in the eventthat the movement amount v differs for each line, k number, equal to thenumber of lines, of the movement amount v is supplied as continuityinformation from the continuity setting unit 15152 to the modelgenerating unit 15161.

Also, processing region information which specifies the processingregion within the input image is also supplied to the model generatingunit 15161 from the processing region setting unit 15151. In the case ofusing a rectangular-shaped region as the processing region, theprocessing region information can be expressed, for example, bycoordinate values at opposite angles of a rectangular-shaped processingregion as to an absolute coordinates system wherein a predeterminedposition (pixel) of the input image is the origin, or the centercoordinate of the processing region and the width (horizontal direction)and the height (vertical direction) of the processing region, or thelike.

The model generating unit 15161 acquires, for example, a processingregion width of l, and k line numbers, as illustrated in FIG. 119, fromthe processing region information supplied from the processing regionsetting unit 15151. In this example, let us say that the processingregion 15042 is rectangular-shaped, and for example, has l=10 and k=4.In this case, the processing region width of each line (the number ofpixels) are the same, but with the processing region 15042, each linecan have a different processing region width. In the case that theprocessing region width l differs for each line, the model generatingunit 15021 acquires k, which is hew same number as the number of linesof the processing region width l.

Also, the model generating unit 15161 generates an equation (modelequation) of a mixed pixel value P_(i) which is shown in theabove-described Expression (141), based on the movement amount vsupplied from the continuity setting unit 15152 and the processingregion width l and k number of lines acquired from the processing regioninformation, and supplies this to the normal equation generating unit15173.

The constraint condition information specified by the user operating theuser I/F 15156 is supplied to the constraint condition setting unit15171 from the user I/F 15156. For example, the user performs anoperation with the user I/F 15156 to select (specify) the desiredconstraint condition from the constraint conditions which constrain therelation of the pixel values Q_(h) of each of the pixels of an imagewith no movement blurring, such as the above-described Expression (143)or Expression (144), which have been set in advance inside theconstraint condition setting unit 15171, and the user I/F 15156 suppliesthe information describing the selected constraint conditions as theconstraint condition information to the constraint condition settingunit 15171.

The constraint condition setting unit 15171 selects a constraintcondition expression corresponding to the constraint conditioninformation supplied from the user I/F 15156 from the constraintcondition expressions within the constraint condition setting unit 15171itself. Further, the constraint condition setting unit 15171 acquiresthe processing region width l based on the processing region informationof the input image supplied to the equation generating unit 15162 fromthe processing region setting unit 15151, and generates only the numberwhich corresponds to the processing region width l, of he constraintcondition expressions. For example, in the event that Expression (143)is used which has the condition “adjacent pixel difference=0” as theconstraint condition information, the constraint condition setting unit15171 generates the constraint condition expression expressed by theabove-described Expression (145), by acquiring the processing regionwidth l based on the processing region information supplied from theprocessing region setting unit 15151, and supplies this to the normalequation generating unit 15173. In the example of the processing region15042 illustrated in FIG. 119, the processing region 15042 isrectangular-shaped, and the processing region width l is the same ofeach line, but in the case that the processing region width l differsfor each line, the number of constraint condition expressions alsodiffer for each line.

The weighting changing unit 15172 has a weighting W_(mi) (hereafter willbe referred to simply as weighting W_(mi)) as to the equation for eachof the mixed pixel values P_(i) (the model equation) and a weightingW_(bj) (hereafter will be referred to simply as weighting W_(bj)) as tothe constraint condition expression. The values for each of theweighting W_(mi) and the weighting W_(bj) are supplied from the user I/F15156 by the user operating the user I/F 15156, and are changed from thedefault value (for example, 1). In other words, the user can decide(set) each of the values for the weighting W_(mi) and the weightingW_(bj), by operating the user I/F 15156. The weighting changing unit15172 supplies each of the weighting W_(mi) and the weighting W_(bj),set by the user, to the normal equation generating unit 15173, and sets(changes) the weighting of the expressions corresponding to each of themodel equation and the constraint condition expression of the normalequation generating unit 15173.

Specifically, with the normal equation generating unit 15173, theweighting W_(bj) is set by the weighting changing unit 15172, as toExpression (147) which considers the error e_(bj) in the constraintcondition Expression (145) which is supplied from the constraintcondition setting unit 15171. In other words, Expression (147) ischanged to Expression (161) which has multiplied both sides thereof bythe weighting W_(bj).

Also, with the normal equation generating unit 15173, the weightingW_(mi) is set by the weighting changing unit 15172, as to Expression(146) which considers the error e_(mi) in the equation for mixed pixelvalue P_(i) (model equation) (141) which is supplied from the modelgenerating unit 15161. In other words, Expression (146) is changed tothe above-described Expression (162) which has multiplied both sidesthereof by the weighting W_(mi).

The normal equation generating unit 15173 selects one predetermined linefrom the processing region 15042 as the line of interest, as illustratedin FIG. 128. Then the normal equation generating unit 15173 generatesthe normal equation expressed in the above-described Expression (163)from Expression (161) and Expression (162), corresponding to theselected line of interest.

Here, Expression (163) is replaced with Expression (164), using thematrix A″ and the column vectors x, y, e″. In this case, the total sumE″ of the squared errors of the error e_(bj) of Expression (161) and theerror e_(mi) of Expression (162) can be expressed with Expression (165),and in order to find the Q_(h) which causes the total sum E″ of thesquared errors to be the minimum, the column vector x (the column vectorx with the pixel value Q_(h) as a component thereof) can be expressedwith Expression (166), similar to the above-described Expressions (150)and (151).

Accordingly, the normal equation generating unit 15173 computes thematrix (A″^(T)A″)⁻¹A″^(T), which is a coefficient of the column vector yof the right side of Expression (166), and substitutes the mixed pixelvalue P_(i) which is acquired based on the input image supplied from thesensor 2 (FIG. 1) into the column vector y. The matrix(A″^(T)A″)⁻¹A″^(T) is computed, and Expression (166) into which themixed pixel value P_(i) is substituted is supplied from the normalequation generating unit 15173 to the actual world waveform estimatingunit 15163.

When the processing such as described above is finished wherein onepredetermined line of the processing region 15042 is selected as theline of interest, the normal equation generating unit 15173 selects aline not yet selected as the next line of interest, and similarprocessing is performed as to the selected next line of interest.

The actual world waveform estimating unit 15163, by computing Expression(166) which is supplied from the normal equation generating unit 15173,finds the column vector x, that is to say, the pixel value Q_(h) with nomovement blurring, and supplies this to the image generating unit 15154(FIG. 147).

The image generating unit 15154 replaces a portion of the processingregion 15042 of the input image with the image of the pixel value Q_(h)with no movement blurring, which is supplied from the actual worldwaveform estimating unit 15163, based on the processing regioninformation supplied from the processing region setting unit 15151, andgenerates this as an output image, and supplies this to the imagedisplay unit 15155.

Next, the actual world estimating processing of the actual worldestimating unit 15153 in step S15218 in FIG. 148 will be described withreference to the flowchart in FIG. 150.

First, in step S15251, the model generating unit 15161 acquires theprocessing region width l and the number of lines k, based on theprocessing region information supplied from the processing regionsetting unit 15151, and the flow proceeds to step S15252.

In step S15252, the model generating unit 15161 acquires k movementamounts v, k being equivalent to the number of lines in the processingregion 15042, from the continuity setting unit 15152, and the flowproceeds to step S15253.

In step S15253, the model generating unit 15161 generates an equationfor the mixed pixel value P_(i) for each line, and the flow proceeds tostep S15254. In other words, the model generating unit 15161 generates knumber of model equations (Expression (141)).

In step S15254, the constraint condition setting unit 15171 selects theconstraint condition expression corresponding to the constraintcondition information supplied from the user I/F 15156, from theconstraint condition expressions existing within the constraintcondition setting unit itself. Also, the constraint condition settingunit 15171 acquires the processing region width l from the processingregion information of the input image which is supplied from theequation generating unit 15162 from the processing region setting unit15151, and generates an constraint condition Expression (Expression(145)) corresponding to the constraint condition information from theuser I/F 15156, which comprises the number of equations corresponding tothe processing region width l, and the flow proceeds to step S15255.

In step S15255, the weighting changing unit 15172 sets the weighting ofeach expression of the mixed pixel value P_(i) which is supplied fromthe model generating unit 15161 (the model equation) and the constraintcondition expression which is supplied from the constraint conditionsetting unit 15171, by supplying the weighting W_(bj) and the weightingW_(mi) which are supplied from the user I/F 15156 to the normal equationgenerating unit 15173, and the flow proceeds to step S15256.

In step S15256, the normal equation generating unit 15173 selects onepredetermined line from the processing region 15042 as the line ofinterest, and the flow proceeds to step S15257. In step S15257, thenormal equation generating unit 15173 generates a normal equationexpressed by Expression (163) (Expression (164)) from the constraintcondition Expression (161) wherein the weighting W_(bj) is changed bythe weighting changing unit 15172 as to step S15255, in Expression (145)supplied from the constraint condition setting unit 15171 and Expression(162) wherein the weighting W_(mi) is changed by the weighting changingunit 15172 in step S12522 as to Expression (146) considering the errore_(mi) of each of the expressions of the mixed pixel value Pi expressedby Expression (141) supplied from the model generating unit 15161, andthe flow proceeds to step S15258.

In step S15258, the normal equation generating unit 15173 computes thematrix (A″^(T)A″)⁻¹A″^(T), which is a coefficient of the column vector yof the right side of Expression (166), and the flow proceeds to stepS15259.

In step S15259, the normal equation generating unit 15173 substitutesthe mixed pixel value P_(i) which is acquired based on the input imagesupplied from the sensor 2 (FIG. 1) into the column vector y, and theflow proceeds to step S15260.

In step S15260, the actual world waveform estimating unit 15163 computesa normal equation. That is to say, the actual world waveform estimatingunit 15163 finds a pixel value Q_(h) with no movement blurring bycomputing Expression (166) which is supplied from the normal equationgenerating unit 15173, and the flow proceeds to step S15261.

In step S15261, determination is made as to whether or not processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042. In the event that it isdetermined that processing is not performed to find a pixel value Q_(h)with no movement blurring as to all lines of the processing region15042, the process is returned to step S15256, and the processing insteps S15256 through S15261 is repeated. That is to say, a line not yetselected of the processing region 15042 is selected as the next line ofinterest, and processing is performed to find a pixel value Q_(h) withno movement blurring.

On the other hand, in the event that it is determined that processing isperformed to find a pixel value Q_(h) with no movement blurring as toall lines of the processing region 15042, the actual world estimatingprocessing ends and returns.

Thus, by generating a normal equation (Expression (163)) as to one linewhich is taken as the line of interest of the processing region 15042,and repeating the processing to find a pixel value Q_(h) with nomovement blurring k number of times, which is the number of lines in theprocessing region 15042, the pixel value Q_(h) with no movement blurringcan be found across the entire region of the processing region 15042.

Here, in the event that the movement amount v and processing regionwidth l for each line of the processing region 15042 are each the same,and that the weighting W_(bj) and the weighting W_(mi) supplied from theuser I/F 15156 to the weighting changing unit 15172 is set (supplied)similarly with each line of the processing region 15042, the matrix A ofExpression (163) becomes the same for each line. Accordingly, the matrix(A″^(T)A″)⁻¹A″^(T) of Expression (166) becomes the same for each line,and so the processing for computing the matrix (A″^(T)A″)⁻¹A″^(T) foreach line can be omitted.

Thus, the actual world estimating processing of the actual worldestimating unit 15153 in step S15218 in FIG. 148, in the event that themovement amount v and processing region width l for each line of theprocessing region 15042 are each the same, and that the weighting W_(bj)and the weighting W_(mi) supplied from the user I/F 15156 to theweighting changing unit 15172 is set (supplied) similarly with each lineof the processing region 15042, will be described with reference to theflowchart in FIG. 151. Now, with regard to the portions in FIG. 151which are similar to the flowchart in FIG. 150, the description thereofwill be omitted as appropriate.

In FIG. 151, the processing of step S15288 corresponding to step S15256in FIG. 150 is not positioned between steps S15285 and S15286, butrather between steps S15287 and S15289. Further, in step S15291, in theevent that it is determined that the processing for finding the pixelvalue Q_(h) without movement blurring is not performed as to all of thelines in the processing region 15042, the processing returns the flow tostep S15288.

In other words, in the event that the movement amount v and processingregion width l for each line of the processing region 15042 are each thesame, and that the weighting W_(bj) and the weighting W_(mi) suppliedfrom the user I/F 15156 to the weighting changing unit 15172 is set(supplied) similarly with each line of the processing region 15042, thenormal equation generated in step S15286 becomes the same for each line,and therefore, after computing the matrix (A″^(T)A″)⁻¹A″^(T) as to thefirst line of interest in step S15287, computing the matrix(A″^(T)A″)⁻¹A″^(T) (the processing in step S15287) as to the line ofinterest hereafter is not necessary. Accordingly, the selection of theline of interest (step S15288), the substitution of the mixed pixelvalue P_(i) (the processing in step S15289), and the computing of thenormal equation (the processing in step S15290) need only be executed inorder, as to the lines in the processing region 15042 which have not yetbeen selected.

Thus, with the embodiment illustrated in FIG. 147, a normal equation isgenerated from the equation of the mixed pixel value P_(i) wherein thepixel value with movement blurring occurring is modeled with a pixelvalue with no movement blurring, and the constraint condition expressionusing the relation between the adjacent pixels of the pixel values withno movement blurring, and by computing this normal equation, a pixelvalue with no movement blurring can be found, and thus an image withgood image quality can be generated.

Also, the weighting for each of the mixed pixel value P_(i) equation andconstraint condition expression is supplied to the signal processingdevice 4 and set as desired values by the user operating the user I/F15156, and therefore, an image of high image quality according to thepreference of the user can be generated.

In the above-described example, the information showing which constraintcondition the user selected (specified) from the several constraintconditions previously set inside the constraint condition setting unit15171 is used as the constraint condition information supplied from theuser I/F 15156 to the constraint condition setting unit 15171, but theuser may also directly input the constraint condition expression byoperating the user I/F 15156.

Also, in addition to the above-described embodiment, with the constraintcondition setting unit 15171, the predetermined constraint conditionsare determined (fixed), the user can operate the user I/F 15156 toselect the constraint conditions, or use an embodiment wherein directlyinputting the constraint condition expression is not necessary.

Now, with the continuity setting unit 15012 in FIG. 113 (similar withother continuity setting units), the movement as the continuityinformation can be set based on the user operation, but also can be setby detecting the movement from the input image.

The detecting method of the movement with the continuity setting unit15012 in FIG. 113 will be described.

In the event that a certain object is moving in an input image, as for amethod for detecting movement vector serving as the movement of theobject for example, the so-called block matching method has been known.

However, with the block matching method, matching is performed betweenthe frame of interest and the frames before and after the frame ofinterest, so that movements cannot be detected easily with the frame ofinterest alone.

To this end, the continuity setting unit 15012 is configured so as todetect movements from an input image having one frame alone.

FIG. 152 illustrates a configuration example of the continuity settingunit 15012 in FIG. 113.

With the continuity setting unit 15012 of which the configuration isshown in FIG. 152, the movement direction of an object in the processingregion of the an input image is detected, and the input image iscorrected such that the movement direction becomes the horizontaldirection. Subsequently, the features subjected to the one-dimensionaldifferentiation in the movement direction of the object in the inputimage, which are the differential value of the pixel values of pixelsadjacent in the movement direction, are detected.

Further, the correlation is detected between the features of the pixelof interest and the features of the corresponding pixel with apredetermined distance in the movement direction, and the movementamount of the object is detected according to the distance between thecorresponding pixel and the pixel of interest, which exhibits themaximum detected correlation.

The continuity setting unit 15012 of which the configuration is shown inFIG. 152 includes a movement direction detecting unit 11201, a movementdirection correcting unit 11202, a features detecting unit 11203, and amovement amount detecting unit 11204.

Further, the movement direction detecting unit 11201 includes anactivity computing unit 11211, and an activity evaluating unit 11212.The movement direction correcting unit 11202 includes an affinetransformation unit 11213.

The features detecting unit 11203 includes a difference computing unit11214, a difference evaluating unit 11215, an intermediate imagecreating unit 11216, an intermediate image creating unit 11217, framememory 11218, a sign inverting unit 11219, and frame memory 11220.

Further, the movement amount detecting unit 11204 includes a correlationdetecting unit 11221, and a correlation evaluating unit 11222.

With the continuity setting unit 15012 wherein the configuration isillustrated is FIG. 152, the input image is supplied to the movementdirection detecting unit 11201 and the movement direction correctingunit 11202. Further, the processing region information that is outputfrom the processing region setting unit 15011 in FIG. 113 is alsosupplied to the movement direction detecting unit 11201 and the movementdirection correcting unit 11202.

The movement direction detecting unit 11201 acquires the input image andthe processing region information, and detects the movement direction inthe processing region from the acquired input image.

When capturing a moving object, movement blurring occurs on the image ofthe object. This is caused by actions of the image sensor of a camera orvideo camera serving as the sensor 2 for capturing the image of anobject.

That is to say, an image sensor such as a CCD (Charge Coupled Device) orCMOS (Complementary Metal-Oxide Semiconductor) sensor consecutivelyconverts the incident light into electric charge for each pixel duringexposure time (shutter time), and further converts the electric chargeinto one pixel value. When an object to be captured is in a stationarystate, the image (light) of the same portion of the object is convertedinto one pixel value during exposure time. The image thus capturedincludes no movement blurring.

On the other hand, when an object is moving, the image of the portion ofthe object to be cast into one pixel changes during exposure time, andthe images of different portions of the object are converted into onepixel value by accident. Speaking inversely, the image of one portion ofthe object is cast into multiple pixel values, which is movementblurring.

Movement blurring occurs in the movement direction of the object.

Upon focusing on the pixel values of pixels arrayed in the movementdirection of the portion where movement blurring occurs (regionincluding movement blurring), the image of generally the same rangeportion of the object is projected to the pixel values of the pixelsarrayed in the movement direction. Accordingly, we can say that changein the pixel values of the pixels arrayed in the movement direction atthe portion where movement blurring occurs is further reduced.

The movement direction detecting unit 11201 detects a movement directionbased on such change, i.e., activity in the pixel value of a pixel inthe processing region of an input image.

More specifically, the activity computing unit 11211 of the movementdirection detecting unit 11201 computes change (activity) in the pixelvalues of pixels arrayed in various directions for each predetermineddirection. For example, the activity computing unit 11211 computes thedifference between the pixel values of the pixels positionedcorresponding to each direction for each predetermine direction asactivity. The activity computing unit 11211 supplies informationindicating change in the computed pixel values to the activityevaluating unit 11212.

The activity evaluating unit 11212 selects the minimum change in thepixel value, of change in the pixel values of the pixels for eachpredetermined direction supplied from the activity computing unit 11211,and takes the direction corresponding to the selected change in thepixel value as the movement direction.

The movement direction detecting unit 11201 supplies movement directioninformation indicating the movement direction thus detected to themovement direction correcting unit 11202.

The movement direction correcting unit 11202 is supplied with processingregion information as well. The movement direction correcting unit 11202converts the image data within the processing region in the input imagebased on the movement direction information supplied from the movementdirection detecting unit 11201 such that the movement direction becomesthe horizontal direction of the image.

For example, the affine transformation unit 11213 of the movementdirection correcting unit 11202 subjects the image data within theprocessing region in the input image to affine transformation based onthe movement direction information supplied from the movement directiondetecting unit 11201 such that the movement direction shown in themovement direction information becomes the horizontal direction of theimage.

The movement direction correcting unit 11202 supplies the image datawithin the processing region in the input image converted such that themovement direction becomes the horizontal direction of the image to thefeatures detecting unit 11203.

Now, with the input image, the hypothesis is that the object is movingin the horizontal direction (from the left to the right direction), andtherefore the continuity setting unit 15012 can be configured withoutproviding the movement direction detecting unit 11201 and the movementdirection correcting unit 11202.

The features detecting unit 11203 detects the features of the imagesupplied from the movement direction correcting unit 11202.

That is to say, the difference computing unit 11214 of the featuresdetecting unit 11203 sets a pixel selected by selecting one pixel fromthe pixels in the processing region of the input image, as a pixel ofinterest. Subsequently, the difference computing unit 11214 of thefeatures detecting unit 11203 obtains a difference value by subtractingthe pixel value of the pixel adjacent to the pixel of interest on theright side from the pixel value of the pixel of interest.

The difference computing unit 11214 obtains difference values by takingthe pixels in the processing region of the input image as a pixel ofinterest in order. That is to say, the difference computing unit 11214obtains difference values regarding all of the pixels in the processingregion of the input image. The difference computing unit 11214 suppliesthe difference values thus computed to the difference evaluating unit11215 along with information indicating the position of the pixel ofinterest corresponding to each obtained difference value (positionalinformation indicating the position on the screen of each differencevalue).

The difference evaluating unit 11215 determines regarding whether or notthe difference values are 0 or more, supplies the difference valuesequal to or greater than 0 to the intermediate image creating unit 11216along with the positional information indicating the position on thescreen of each difference value, and supplies the difference values lessthan 0 to the intermediate image creating unit 11217 along with thepositional information indicating the position on the screen of eachdifference value.

The intermediate image creating unit 11216 creates, based on thedifference values equal to or greater than 0 supplied from thedifference evaluating unit 11215 along with the positional informationindicating the position on the screen of the difference values, anintermediate image made up of each difference value. That is to say, theintermediate image creating unit 11216 creates an intermediate image bysetting the difference values equal to or greater than 0 supplied fromthe difference evaluating unit 11215 to the pixels at the positions onthe screen indicated by the positional information, and setting 0 to thepixels at the positions where no difference value is supplied from thedifference evaluating unit 11215. The intermediate image creating unit11216 supplies the intermediate image thus created (hereafter, referredto as non-inverted intermediate image) to the frame memory 11218.

The intermediate image creating unit 11217 creates, based on thedifference values less than 0 (negative values) supplied from thedifference evaluating unit 11215 along with the positional informationindicating the positions on the screen of the difference values, anintermediate image made up of the difference values. That is to say, theintermediate image creating unit 11217 creates an intermediate image bysetting the difference values less than 0 supplied from the differenceevaluating unit 11215 to the pixels at the positions on the screenindicated by the positional information, and setting 0 to the pixels atthe positions where no difference value is supplied from the differenceevaluating unit 11215. The intermediate image creating unit 11216supplies the intermediate image thus created to the sign inverting unit11219.

The sign inverting unit 11219 inverts the signs of the difference valuesless than 0 set to the pixels of the intermediate image supplied fromthe intermediate image creating unit 11217. The signs of the value 0 setto the pixels of the intermediate image are not inverted. That is tosay, the sign inverting unit 11219 selects the difference values lessthan 0 set to the pixels of the intermediate image supplied from theintermediate image creating unit 11217, and converts the selecteddifference values less than 0 into the values greater than 0 having thesame absolute values as the difference values. For example, thedifference value, which is −15, is converted into 15 by inverting thesign thereof. The sign inverting unit 11219 supplies the intermediateimage thus sign-inverted (hereafter referred to as inverted intermediateimage) to the frame memory 11220.

The frame memory 11218 supplies the non-inverted intermediate image madeup of the difference values equal to or greater than 0 and 0 to themovement amount detecting unit 11204 as features. The frame memory 11220supplies the inverted intermediate image made up of the differencevalues greater than 0 of which the signs are inverted and 0 to themovement amount detecting unit 11204 as features.

The movement detecting unit 11204 detects movements based on thefeatures supplied from the features detecting unit 11203. That is tosay, the movement detecting unit 11204 detects the correlation betweenthe features of at least the pixel of interest, of the pixels of theimage of the object in the processing region of the input image, and thefeatures of the corresponding pixel allocated in the movement directionas to the pixel of interest, and detects the movement amount of theimage of the object in the processing region of the input imageaccording to the detected correlation.

The correlation detecting unit 11221 of the movement amount detectingunit 11204 detects the correlation between the non-inverted intermediateimage serving as features supplied from the frame memory 11218 of thefeatures detecting unit 11203, and the inverted intermediate imageserving as features supplied from the frame memory 11220 of the featuresdetecting unit 11203. The correlation detecting unit 11221 supplies thedetected correlation to the correlation evaluating unit 11222.

More specifically describing, for example, the correlation detectingunit 11221 of the movement amount detecting unit 11204 moves (shifts)the inverted intermediate image made up of the difference values ofwhich the signs are inverted so as to be greater than 0, and 0 suppliedfrom the frame memory 11220 of the features detecting unit 11203, in thehorizontal direction of the screen in units of pixel as to thenon-inverted intermediate image made up of the difference values equalto or greater than 0, and 0 supplied from the frame memory 11218 of thefeatures detecting unit 11203. That is to say, the correlation detectingunit 11221 moves the positions on the screen of the pixels making up theinverted intermediate image in the horizontal direction.

The positional relation on the screen between the pixels of thenon-inverted intermediate image and the pixels of the invertedintermediate image changes by moving (the pixels of) the invertedintermediate image in the horizontal direction on the screen. Forexample, the corresponding pixel of the inverted intermediate imagepositioned on the screen corresponding to the pixel of interest of thenon-inverted intermediate image before movement results in departingfrom the position corresponding to the pixel of interest of thenon-inverted intermediate image by the movement distance after movement.More specifically, when the non-inverted intermediate image is moved tothe right by 20 pixels, the corresponding pixel of the invertedintermediate image departs from the position corresponding to the pixelof interest of the non-inverted intermediate image to the right by 20pixels. Speaking inversely, the corresponding pixel of the invertedintermediate image positioned on the screen corresponding to the pixelof interest of the non-inverted intermediate image after movementdeparts from the position corresponding to the pixel of interest by themovement distance before movement.

The correlation detecting unit 11221 computes the difference between thepixel values of the pixels corresponding to the non-invertedintermediate image and the inverted intermediate image moved, and takesthe sum of the differential absolute values as a correlation value.

For example, the correlation detecting unit 11221 moves (shifts) theinverted intermediate image in the horizontal direction of the screen inincrements of one pixel in a range of 70 pixels in the left direction ofthe screen through 70 pixels in the right direction of the screen as tothe non-inverted intermediate image, computes the difference between thepixel values of the pixels to be positioned at the same position on thescreen regarding the non-inverted intermediate image and the invertedintermediate image moved for each moved position (each movementdistance), and takes the sum of the differential absolute values as acorrelation value.

For example, when the inverted intermediate image is moved to the leftdirection of the screen as to the non-inverted intermediate image, themovement distance is represented with a negative (minus). When theinverted intermediate image is moved to the right direction of thescreen as to the non-inverted intermediate image, the movement distanceis represented with a positive (plus). The correlation detecting unit11221 computes the difference between the pixel values of the pixels tobe positioned at the same position on the screen regarding thenon-inverted intermediate image and the inverted intermediate imagemoved for each movement distance of −70 pixels through +70 pixels, andtakes the sum of the differential absolute values as a correlationvalue.

The correlation detecting unit 11221 supplies the correlation valuecorresponding to the movement distance to the correlation evaluatingunit 11222. That is to say, the correlation detecting unit 11221supplies a pair of the movement distance and the correlation value tothe correlation evaluating unit 11222.

The correlation evaluating unit 11222 detects the movement amount of theimage of the object in the processing region of the input imageaccording to the correlation. More specifically, the correlationevaluating unit 11222 takes, of the correlations supplied from thecorrelation detecting unit 11221, the movement distance corresponding tothe maximum (strongest) correlation as a movement amount.

For example, the correlation evaluating unit 11222 selects the minimumvalue, of the sum of the differential absolute values serving as thecorrelation value supplied from the correlation detecting unit 11221,and sets the movement distance corresponding to the selected minimumvalue to the movement amount.

The correlation evaluating unit 11222 outputs the detected movementamount.

FIG. 153 through FIG. 155 are diagrams for describing the principle fordetecting movements by the continuity setting unit 15012 in FIG. 152.

Now, let us say that a white foreground object serving as an object tobe captured is disposed in front of a black background object serving asanother object to be captured, and is moving from the left side to theright side, and a camera having an image sensor such as a CCD or CMOSsensor captures the foreground object along with the background objectwith a predetermined exposure time (shutter time).

In this case, upon focusing on one frame of the image to be output fromthe camera, the background object is black, so that the camera outputs,for example, a pixel value 0 as to the background object image. Theforeground object is white, so that the camera outputs, for example, apixel value 255 as to the foreground object image. Note that here, letus say that the camera outputs a pixel value in a range of 0 through2⁸−1.

The diagram in the upper side of FIG. 153 is a diagram illustrating thepixel values of the image to be output by the camera when the foregroundobject is in a stationary state at the position at the moment that theshutter of the camera opens (moment of starting exposure).

The diagram in the lower side of FIG. 153 is a diagram illustrating thepixel values of the image to be output by the camera when the foregroundobject is in a stationary state at the position at the moment that theshutter of the camera closes (moment of ending exposure).

As shown in FIG. 153, the movement amount of the image of the foregroundobject is a distance wherein the image of the foreground object movesfrom the moment that the shutter of the camera opens until the momentthat the shutter of the camera closes.

FIG. 154 is a diagram illustrating the pixel values of the image to beoutput from the camera when the foreground object moving in front of thebackground object is captured by the camera. The image sensor of thecamera consecutively converts the image (light) of the object intoelectric charge during exposure time (shutter time) for each pixel, andfurther converts the electric charge into one pixel value, andaccordingly, the image of the foreground object 11251 is projected intothe pixel values of the multiple pixels. The maximum pixel value of theimage shown in FIG. 154 is small as compared with the maximum pixelvalue of the image shown in FIG. 153.

The slope width of the pixel values shown in FIG. 154 corresponds to thewidth of the image of the background object.

Upon the difference value as to the adjacent pixel on the right sidebeing computed, and set to the pixel regarding each pixel of the imageshown in FIG. 154, the image made up of the difference values shown inFIG. 155 is obtained.

That is to say, one pixel is selected from the pixels of the image shownin FIG. 154, and set to as a pixel of interest to which attention ispaid. Subsequently, the difference value is obtained by subtracting thepixel value of the pixel adjacent to the pixel of interest on the rightside from the pixel value of the pixel of interest. The difference valueis set to the pixel at the position corresponding to the pixel ofinterest. The pixels of the image shown in FIG. 154 are taken as a pixelof interest in order, and the image made up of the difference valuesshown in FIG. 155 is obtained.

The difference value having a negative (minus) sign emerges on theone-pixel left side as to the position of the foreground object at themoment that the shutter of the camera opens shown in the diagram on theupper side of FIG. 153, and the difference value having a positive(plus) sign emerges on the one-pixel left side as to the position of theforeground object at the moment that the shutter of the camera closesshown in the diagram on the lower side of FIG. 153.

Accordingly, upon matching being performed between a value obtained byinverting the sign of the difference value having a negative (minus)sign and the difference value having a positive (plus) sign shown inFIG. 155, the movement distance of the value obtained by inverting thesign of the difference value having a negative (minus) sign is the sameas the movement amount on the basis of the difference value having apositive (plus) sign when performing matching, for example.

For example, on the basis of the difference value having a positive(plus) sign, the value obtained by inverting the sign of the differencevalue having a negative (minus) sign is moved in the horizontaldirection, the correlation between the value obtained by inverting thenegative difference value and the positive difference value is detectedfor each movement distance thereof, thereby detecting the maximum(strongest) correlation. The movement distance when the maximumcorrelation is detected is the same as the movement amount.

More specifically, for example, on the basis of the difference valuehaving a positive (plus) sign, the value obtained by inverting the signof the difference value having a negative (minus) sign is moved in thehorizontal direction, as the correlation between the value obtained byinverting the negative difference value and the positive differencevalue for each movement distance thereof, the positive difference valueis subtracted from the inverted value for each pixel. Subsequently, theminimum value within the subtracted results, i.e., the maximumcorrelation is detected. The movement distance corresponding to thedetected maximum correlation is the same as the movement amount.

As described above, the movement amount, which is the amount wherein theimage of the object moves, can be detected from one frame of the imageduring exposure time (shutter time).

That is to say, the difference computing unit 11214 of the featuresdetecting unit 11203 selects one pixel from pixels in the processingregion of the input image, sets this as a pixel of interest, andsubtracts the pixel value of the pixel adjacent to the pixel of intereston the right side from the pixel value of the pixel of interest, therebycomputing the difference values shown in FIG. 155, for example. Thedifference evaluating unit 11215 classifies the difference values intothe positive difference values and the negative difference values basedon the signs of the difference values.

The intermediate image creating unit 11216 creates, from the classifiedpositive difference values, a non-inverted intermediate image made up ofthe positive difference values thereof. The intermediate image creatingunit 11217 creates, from the classified negative difference values, anintermediate image made up of the negative difference values thereof.The sign inverting unit 11219 creates a non-inverted intermediate imageby inverting the signs of the negative pixel values of the intermediateimage made up of the negative difference values.

The movement amount detecting unit 11204 obtains the movement distanceof the non-inverted intermediate image and the inverted intermediateimage, which have the strongest correlation, and takes the obtainedmovement distance as the movement amount.

When the features detecting unit 11203 detects the image of the movingobject, and detects the features of the image of the moving object, themovement amount detecting unit 11204 detects a correlation based on thefeatures, and detects the movement amount of the image of the objectwithin the input image according to the detected correlation.

Also, when the features amount detecting unit 11203 selects the pixel ofinterest to which attention is paid from the pixels belonged to theimage of the moving object, and detects the features of the pixel ofinterest, the movement amount detecting unit 11204 detects thecorrelation between the features of the pixel of interest and thefeatures of the corresponding pixel to be allocated in the movementdirection as to the pixel of interest, and detects the movement amountof the image of the object in the processing region of the input imageaccording to the detected correlation.

FIG. 156 is a flowchart for describing the processing for detecting amovement amount by the continuity setting unit 15012 in FIG. 152.

In step S11201, the movement direction detecting unit 11201 and themovement direction correcting unit 11202 acquire the input image and theprocessing region information, and the flow proceeds to step S11202.

In step S11202, the activity computing unit 11211 of the movementdirection detecting unit 112011 computes activity regarding the pixelsof the processing region in the input image acquired in the processingin step S11201, and the flow proceeds to step S11203.

For example, the activity computing unit 11211 selects the pixel ofinterest to which attention is paid, of the pixels of the processingregion in the input image. The activity computing unit 11211 extracts apredetermined number of perimeter pixels in the vicinity of the pixel ofinterest. For example, the activity computing unit 11211 extractsperimeter pixels made up of 5×5 pixels (5 pixels by 5 pixels) centeredon the pixel of interest.

Subsequently, the activity computing unit 11211 detects activitycorresponding to the predetermined direction on the image from theextracted perimeter pixels.

With the following description, a pixel array in the horizontaldirection is referred to as a row, and a pixel array in the verticaldirection is referred to as a column.

Regarding 5×5 perimeter pixels for example, the activity computing unit11211 detects activity as to the angle of 90 degrees (vertical directionof the screen) on the basis of the horizontal direction of the screen bycomputing the differences between the pixel values of the adjacentpixels in the up-and-down (vertical) direction on the screen, dividingthe sum of the differential absolute values computed by the number ofdifferences, and taking the result as activity.

For example, the difference of pixel values is computed regarding thetwo pixels adjacent in the up-to-down direction on the screen of 20pairs, the sum of the differential absolute values computed is dividedby 20, and the result (quotient) is set to the activity as to the angleof 90 degrees.

Regarding 5×5 perimeter pixels for example, the activity computing unit11211 detects activity as to the angle of 76 degrees (tan⁻¹ (4/1)) onthe basis of the horizontal direction of the screen by computing thedifferences between the respective pixel values of the leftmost pixelthrough the fourth pixel from the left in the lowest row, and therespective pixel values of the pixels on the four-pixel upper side andalso on one-pixel right side as to the respective pixels, dividing thesum of the differential absolute values computed by the number ofdifferences, and taking the result as activity.

For example, the difference of pixel values is computed regarding thetwo pixels in the upper right direction having a distance of four pixelsin the vertical direction, and one pixel in the horizontal direction offour pairs, the sum of the differential absolute values computed isdivided by four, and the result (quotient) is set to the activity as tothe angle of 76 degrees.

The activity computing unit 11211 detects activity as to the angle in arange of 90 degrees through 180 degrees on the basis of the horizontaldirection of the screen with the same processing. In the event ofdetecting activity as to the angle in a range of 90 degrees through 180degrees, activity is calculated based on the difference of the pixelvalues of the pixels positioned in the upper left direction.

The activity thus detected is taken as the activity as to the pixel ofinterest.

Note that the detected activity may be the activity as to the perimeterpixels.

Also, description has been made that the perimeter pixels are made up of5×5 pixels (5 pixels by 5 pixels), but pixels having a desired range maybe employed regardless of 5×5 pixels. In the event of employing a greatnumber of perimeter pixels, angular resolution improves.

The activity computing unit 11211 supplies information indicatingactivity corresponding to multiple directions to the activity evaluatingunit 11212.

Returning to FIG. 156, in step S11203, the activity evaluating unit11212 obtains the movement direction by selecting the minimum activitybased on the activity corresponding to a predetermined directioncalculated in the processing in step S11202, and taking the selecteddirection as the movement direction, and the flow proceeds to stepS11204.

In step S11204, the movement direction correcting unit 11202 convertsthe image data in the processing region of the input image based on themovement direction obtained in the processing in step S11203 such thatthe movement direction becomes the horizontal direction of the image,and the flow proceeds to step S11205. For example, in step S11204, theaffine transformation unit 11213 of the movement direction correctingunit 11202 subjects the image data in the processing region of the inputimage to affine transformation based on the movement direction obtainedin the processing in step S11203 such that the movement directionbecomes the horizontal direction of the image. More specifically, forexample, the affine transformation unit 11213 subjects the image data inthe processing region of the input image to affine transformation so asto rotate 18degrees in the clockwise direction on the basis of thehorizontal direction of the screen when the movement direction is theangle of 18 degrees.

In step S11205, the difference computing unit 11214 of the featuresdetecting unit 11203 computes the difference value of the pixel value asto the pixel adjacent in the horizontal direction regarding each pixelin the processing region of the input image converted such that themovement direction becomes the horizontal direction of the screen in theprocessing in step S11204, and the flow proceeds to step S11206.

For example, in step S11205, the difference computing unit 11214 sets apixel of interest to which attention is paid by selecting one pixel fromthe pixels in the processing region of the input image. Subsequently,the difference computing unit 11214 obtains the difference value bysubtracting the pixel value of the pixel adjacent to the pixel ofinterest on the right side from the pixel value of the pixel ofinterest.

In step S11206, the difference evaluating unit 11215 of the featuresdetecting unit 11203 allocates the difference values based on the signsof the difference values, and the flow proceeds to step S11207. That isto say, the difference evaluating unit 11215 supplies the differencevalues equal to or greater than 0 to the intermediate image creatingunit 11216, and supplies the difference values less than 0 to theintermediate image creating unit 11217. In this case, the differenceevaluating unit 11215 supplies the difference values to the intermediateimage creating unit 11216 or the intermediate image creating unit 11217along with the positional information indicating the position of eachdifference value on the screen.

In step S11207, the intermediate image creating unit 11216 of thefeatures detecting unit 11203 creates an intermediate image made up ofthe positive difference values based on the difference values equal toor greater than 0 (positive difference values) allocated in theprocessing in step S11206, and the flow proceeds to step S11208. That isto say, in step S11207, the intermediate image creating unit 11216creates an intermediate image by setting the positive difference valuesto the pixels at the positions on the screen indicated with thepositional information, and setting 0 to the pixels at the positionswhere no difference value was supplied.

Thus, a non-inverted intermediate image is created in the processing instep S11207. In step S11208, the intermediate image creating unit 11217of the features detecting unit 11203 creates an intermediate image madeup of the negative difference values based on the difference values lessthan 0 (negative difference values) allocated in the processing in stepS11206, and the flow proceeds to step S11209. That is to say, in stepS11208, the intermediate image creating unit 11217 creates anintermediate image by setting the negative difference values to thepixels at the positions on the screen indicated with the positionalinformation, and setting 0 to the pixels at the positions where nodifference value was supplied.

In step S11209, the sign inverting unit 11219 of the features detectingunit 11203 inverts the signs of the negative difference values of theintermediate image made up of the negative difference values created inthe processing in step S11208. That is to say, in step S11209, thenegative difference values set to the pixels of the negativeintermediate image are converted into the positive values having thesame absolute values.

Thus, in step S11209, a non-inverted intermediate image is created, andthen the flow proceeds to step S11210.

In step S11210, the movement amount detecting unit 11204 executescorrelation detecting processing. The details of the processing in stepS11210 will be described later with reference to the flowchart in FIG.157.

In step S11211, the correlation evaluating unit 11222 selects thestrongest correlation, of the correlations detected in the processing instep S11210, and the flow proceeds to step S11212. For example, in stepS11211, of the correlation values serving as the sum of the differentialabsolute values of the pixel values, the minimum correlation value isselected.

In step S11212, the correlation evaluating unit 11222 sets the movementdistance corresponding to the strongest correlation selected in theprocessing in step S11211 to the movement amount, and the flow proceedsto step S11213. For example, in step S11212, of the correlation valuesserving as the sum of the differential absolute values of the pixelvalues, the movement distance of the inverted intermediate image storedin the processing in step S11223 described later corresponding to theselected minimum correlation value is set to the movement amount.

In step S11213, the movement amount detecting unit 11204 outputs themovement amount detected in the processing in step S11210, and theprocessing ends.

FIG. 157 is a flowchart for describing correlation detecting processingcorresponding to the processing in step S11210.

In step S11221, the correlation detecting unit 11221 of the movementamount detecting unit 11204 moves the positions of the pixels of theinverted intermediate image created in the processing in step S11209 inthe horizontal direction in increments of pixel, and the flow proceedsto step S11222.

In step S11222, the correlation detecting unit 11221 detects thecorrelation between the non-inverted intermediate image and the invertedintermediate image of which the positions of the pixels are moved in theprocessing in step S11221, and the flow proceeds to step S11223. Forexample, in step S11222, the differences are computed between the pixelvalues of the pixels of the non-inverted intermediate image and thepixel values of the pixels of the inverted intermediate image at thecorresponding positions on the screen, and the sum of the computeddifferential absolute values is detected as a correlation value. Thecorrelation detecting unit 11221 supplies correlation informationindicating the detected correlation to the correlation evaluating unit11222 along with the movement distance of the pixels of the invertedintermediate image in the processing in step S11221.

In step S11223, the correlation evaluating unit 11222 stores thecorrelation detected in the processing in step S11222 along with themovement distance of the pixels of the inverted intermediate image inthe processing in step S11221, and the flow proceeds to step S11224. Forexample, the correlation evaluating unit 11222 stores the correlationvalue serving as the sum of the differential absolute values of thepixel values along with the movement distance of the pixels of theinverted intermediate image in the processing in step S11221.

In step S11224, the correlation detecting unit 11221 determinesregarding whether or not correlation as to all of the movement distanceshas been detected, and in the event that determination is made thatcorrelation has not been detected with some movement distances, the flowreturns to step S11221, where the processing for detecting correlationas to the next movement distance is repeated.

For example, in step S11224, the correlation detecting unit 11221determines whether or not all correlations have been detected in a caseof moving pixels of the inverted intermediate image in the range of 70pixels in the left direction in the image through 70 pixels in the rightdirection in the image.

In the event that determination is made in step S11224 that thecorrelations for all movements amount have been detected, the processingends (returns).

Thus, the correlation detecting unit 11221 can detect correlation.

As described above, the continuity setting unit 15012 of which theconfiguration is shown in FIG. 152 can detect movement amount from oneframe in an image.

Now, while movement has been detected here with regard to a processingregion, an arrangement may be made wherein movement over the entirescreen due to hand shaking, for example, can be detected, by processingthe entire screen.

Also, even in the event that there are a great amount of repetitivepatterns of the same design in the input image, these can be detected inan accurate manner so long as the movement distance and movementdirection of the processing region in the input image to be processed isconstant.

While movement amount detection from one frame in an image has beendescribed above, it is needless to say that an arrangement may be madewherein movement amount is detected from one field.

Also, an arrangement may be made wherein the movement amount is detectedregarding only the perimeter of the selected pixel of interest.

The above-described embodiments include, besides embodiments of theinvention described in the Claims, embodiments regarding the first andsecond signal processing devices given below.

A first signal processing device comprises: processing region settingmeans for setting the processing region within data, wherein lightsignals of the real world have been projected on a plurality of pixelseach having time integration effects and a part of the continuity of thereal world light signals has been lost; movement vector setting meansfor setting movement vectors of an object within the image datacorresponding to the continuity of the real world light signals of whicha part of the continuity has been lost in the image data; modelgenerating means for modeling the region between pixel values of each ofthe pixels within the processing region and pixel values of each of thepixels where there is no movement blurring, with the understanding thatthe pixel values of each of the pixels within the processing region arevalues obtained by pixel values of each of the pixels without movementblurring as to the object being integrated while moving corresponding toa movement vector; normal equation generating means for generatingnormal equations from a first equation wherein the pixel values of thepixels within the processing region have been substituted into a modelgenerated by the model generating means, and a second equation forconstraining the relation between the pixels where there is no movementblurring; activity detecting means for detecting the activitycorresponding to each of the pixels within the processing region; weightchanging means for changing the weighting as to a part of the first andsecond equations according to the detection results by the activitydetecting means; and actual world estimating means for estimating thepixel values of each of the pixels where there is no movement blurring,by computing the normal equations generated by the normal equationgenerating means and of which the weighting has been changed by theweight changing means.

With the second signal processing device, in addition to the features ofthe first signal processing device, the weighting changing means changethe weighting as to a portion of the first and second equations,corresponding to the pixel of interest, to zero, when the activitycorresponding to the pixel of interest within the processing region ofthe image data is greater than the predetermined threshold value.

The above-described embodiments also include embodiments regardingsignal processing methods, programs, and recording mediums, given below.

A signal processing method, program, and recording medium, comprise: aprocessing region setting step for setting the processing region withindata, wherein light signals of the real world have been projected on aplurality of pixels each having time integration effects and a part ofthe continuity of the real world light signals has been lost; a movementvector setting step for setting movement vectors of an object within theimage data corresponding to the continuity of the real world lightsignals of which a part of the continuity has been lost in the imagedata; a model generating step for modeling the region between pixelvalues of each of the pixels within the processing region and pixelvalues of each of the pixels where there is no movement blurring, withthe understanding that the pixel values of each of the pixels within theprocessing region are values obtained by pixel values of each of thepixels without movement blurring as to the object being integrated whilemoving corresponding to a movement vector; a normal equation generatingstep for generating normal equations from a first equation wherein thepixel values of the pixels within the processing region have beensubstituted into a model generated in the model generating step, and asecond equation for constraining the relation between the pixels wherethere is no movement blurring; an activity detecting step for detectingthe activity corresponding to each of the pixels within the processingregion; a weight changing step for changing the weighting as to a partof the first and second equations according to the detection results inthe activity detecting step; and an actual world estimating step forestimating the pixel values of each of the pixels where there is nomovement blurring, by computing the normal equations generated by thenormal equation generating step and of which the weighting has beenchanged in the weight changing step.

INDUSTRIAL APPLICABILITY

As described above, according to the present invention, images and thelike closer approximating real world signals can be obtained.

1. A signal processing device comprising: a processing region settingunit configured to set a processing region within image data wherein alight signal of a real world is projected on a plurality of pixels, eachhaving a time integration effect, and a portion of a continuity of thelight signal of the real world is lost; a movement vector setting unitconfigured to set movement vectors for an object within said image datacorresponding to the continuity of the light signal of the real world,wherein a portion of the continuity thereof of said image data is lost;a model generator configured to model a relation between a pixel valueof each of the pixels within said processing region and a pixel value ofeach of the pixels without movement blurring occurring, assuming thatthe pixel values of each of the pixels within said processing region isa value wherein the pixel value of each of the pixels without saidmovement blurring occurring which correspond to said object isintegrated while shifting corresponding to said movement vectors; anormal equation generator configured to generate a normal equation usinga first equation wherein the pixel value of each of the pixels withinsaid processing region is substituted into a model generated by saidmodel generator, and a second equation which constrains the relationbetween each of the pixels without said movement blurring occurring; anactivity detector configured to detect activities corresponding to eachof the pixels within said processing region; a weighting change unitconfigured to change a weighting of a portion of said first or saidsecond equation, according to detection results of said activitydetector, said weighting change unit configured to change the weightingof a portion of said first and second equations as to a pixel ofinterest to zero when activity corresponding to a pixel of interestwithin said processing region of said image data is at or above apredetermined threshold; and a real world estimating unit configured toestimate the pixel value for each pixel wherein said movement blurringis not occurring, by computing said normal equation which is generatedby said normal equation generator, and wherein the weighting thereof ischanged by said weighting change unit.
 2. A signal processing methodcomprising: using a processor to perform the following; setting aprocessing region within image data wherein a light signal of a realworld is projected on a plurality of pixels, each having a timeintegration effect, and a portion of a continuity of the light signal ofthe real world is lost; setting movement vectors for an object withinsaid image data corresponding to the continuity of the light signal ofthe real world, wherein a portion of the continuity of said image datais lost; generating a model of a relation between a pixel value of eachof the pixels within said processing region and a pixel value of each ofthe pixels without said movement blurring occurring, assuming that thepixel value of each of the pixels within said processing region is avalue wherein the pixel value of each of the pixels without saidmovement blurring occurring which correspond to said object isintegrated while shifting corresponding to said movement vectors;generating a normal equation using a first equation wherein the pixelvalue of each of the pixels within said processing region is substitutedinto the model generated by said generating a model, and a secondequation which constrains the relation between each of the pixelswithout said movement blurring occurring; detecting activitiescorresponding to each of the pixels within said processing region;changing a weighting of a portion of said first or said second equation,according to detection results from said detecting, said changingincluding changing the weighting of a portion of said first and secondequations as to a pixel of interest to zero when activity correspondingto a pixel of interest within said processing region of said image datais at or above a predetermined threshold; and estimating a pixel valueof each pixel wherein said movement blurring is not occurring, bycomputing said normal equation which is generated by said generating anormal equation, and wherein the weighting thereof is changed by saidchanging.
 3. A computer-readable recording medium including computerexecutable instructions, wherein the instructions, when executed by acomputer, cause the computer to perform a method comprising: setting aprocessing region within image data wherein a light signal of a realworld is projected on a plurality of pixels, each having a timeintegration effect, and a portion of a continuity of the light signal ofthe real world is lost; setting movement vectors for an object withinsaid image data corresponding to the continuity of the light signal ofthe real world, wherein a portion of the continuity of said image datais lost; generating a model of a relation between a pixel value of eachof the pixels within said processing region and a pixel value of each ofthe pixels without movement blurring occurring, assuming that the pixelvalue of each of the pixels within said processing region is a valuewherein the pixel value of each of the pixels without said movementblurring occurring which correspond to said object is integrated whileshifting corresponding to said movement vectors; generating a normalequation using a first equation wherein the pixel value of each of thepixels within said processing region is substituted into the modelgenerated by the generating a model, and a second equation whichconstrains the relation between each of the pixels without said movementblurring occurring; detecting activities corresponding to each of thepixels within said processing region; changing a weighting of a portionof said first or said second equation, according to detection resultsfrom said activity detecting, said changing including changing theweighting of a portion of said first and second equations as to a pixelof interest to zero when activity corresponding to a pixel of interestwithin said processing region of said image data is at or above apredetermined threshold; and estimating a pixel value of each pixelwherein said movement blurring is not occurring, by computing saidnormal equation which is generated by said generating a normal equation,and wherein the weighting thereof is changed in said changing.